[R-meta] A rather general question study/effect size ratio
Oops, I forgot to add and ask this, the conceptual conflict in mind is that: 1- On the one hand, the more the studies vs. effects within studies, the higher the generalizability of our meta-analytic results. 2- On the other hand, studies with more effects can influence the studies with a single or fewer effects. It seems to me 1- and 2- are at odds with each other meaning that: A collection of studies each with a single effect gives us the most precise meta-analytic estimates (i.e., rma() type model). Yet, the existence of single-effect studies in a group of multiple-effect studies reduces the precision of our meta-analytic estimates such that multiple-effect studies should influence single-effect studies to improve the precision of the meta-analytic estimates (i.e., rma.mv() type model)
On Fri, Feb 4, 2022 at 12:10 PM Farzad Keyhan <f.keyhaniha at gmail.com> wrote:
Good point, Michael. However, I was thinking of this more from the perspective of sampling. The existence of multiple effects per study (scenarios A & B) may be taken to indicate that the estimates have been sampled under a multistage sampling plan from a larger population of studies (i.e., first, studies were randomly sampled, then, from within them, some effects were randomly sampled, under a 3-level MLMA model). The existence of a single effect per study (scenario C) may be taken to indicate that the estimates have been sampled under a simple random sampling plan from a larger population of studies (i.e., effects were equally likely to be randomly sampled from any study in the population of studies). It would seem to me that generally the more the studies the better our overall sense of the overall effect in the full population of studies. Of the three scenarios (C) is the trickiest to me. It's a tough call to say because each study has provided a single effect, then effects were equally likely to be randomly sampled from any study in the population of studies. Maybe if that had been the case, we would have seen some studies with more than a single effect. Perhaps all of this takes us back to using some form of robust/sandwich estimation of effects, even if we have a single effect per study. Fred On Fri, Feb 4, 2022 at 11:19 AM Michael Dewey <lists at dewey.myzen.co.uk> wrote:
Dear Farzad If having more studies means that the treatment has been tried in a wider range of settings then my vote is for more studies rather than more effects per study. If the settings are homogeneous but the outcomes are varied then more outcomes might improve generalisability. Michael On 04/02/2022 16:50, Farzad Keyhan wrote:
Hello All, All else equal, would it be more desirable to have (A) 40 studies each with two estimates of effect size or (B) 20 studies each with 4 estimates of effect size? My intuition is that (A) is more desirable than (B). Would it also be more desirable to have (C) 80 studies each with 1 estimate of effect size over (A) and (B)? ps. By "desirable" I mean higher generalizability to the target population. Thank you, Fred
_______________________________________________ R-sig-meta-analysis mailing list R-sig-meta-analysis at r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
-- Michael http://www.dewey.myzen.co.uk/home.html