[R-meta] (Too) Many effect sizes for one single group
Hi Catia, I don't think there's a general answer here. It really depends on the specific constructs you're studying and on the details of the studies in your synthesis. That said, I'll keep speculating: If the only difference between the paper that contributes most of the effect size (what I was calling Paper A) and the remaining papers is that Paper A includes multiple replications, then I would probably be more comfortable including all of the effect sizes from Paper A. But often in practice, when you've got a multiple-study paper, the different experiments reported in the paper are testing different variations of the intervention or different tweaks to the protocol. If that's the case, it really depends on whether those variations are also examined in some of the non-Paper A effect sizes. James On Mon, Jan 10, 2022 at 10:41 AM C?tia Ferreira De Oliveira
<cmfo500 at york.ac.uk> wrote:
Dear James, Just to clarify, would this still be the case even if this one paper contained 4 experiments all on different participants? This paper would still be the one mostly driving the nested and lack of independence in the effect sizes. Thank you! I may limit meta-regressions on only a very few number of predictors then. Best wishes, Catia On Mon, 10 Jan 2022 at 15:40, James Pustejovsky <jepusto at gmail.com> wrote:
Hi Catia, Too add to the discussion, the data structure that you've described is one where I think some additional caution is warranted. If half of the effect size estimates in the synthesis come from a single paper (call it Paper A), whereas most of the other studies contribute just one or two effect sizes, then it makes me wonder whether there may be something qualitatively distinctive about Paper A. Are all of the effect sizes in Paper A equally comparable to the effect sizes from the other papers? Or are the effect sizes in Paper A instead covering sources of heterogeneity that are unexplored in the other papers? An example might make my concern a little bit clearer. Say that you're studying a phenomenon where there are several different scales for operationalizing the outcome. Say that you've got 21 studies. In 20 of them, the effect sizes are based one of two different outcome scales. But in the 21st study (Paper A), the investigators measured both of the commonly used outcome scales as well as 10 other scales that all purport to measure the construct. So with this Paper A, the effect sizes are heterogeneous in a way that we don't see in any of the other studies. I think there's a reasonable argument here that the meta-analysis should be conducted by first discarding the uncommon outcome scales from Paper A, and including only the two common scales. The scope of generalization is then more limited because you're looking only at those two scales. But generalization to other outcome scales seems very tenuous because there's really only one study that provides evidence about heterogeneity across outcomes. James On Wed, Jan 5, 2022 at 1:33 PM C?tia Ferreira De Oliveira <cmfo500 at york.ac.uk> wrote:
Dear Wolfgang,
I hope you had a lovely start to the year.
I am sorry for starting the year with questions, but I just wanted to check
whether there is any drawback from including a lot of effect sizes from a
single paper when most labs contributed to the meta-analysis with just one
or two effect sizes? This resulted in a dataset where half of the effect
sizes come from multiple experiments run by the same group. The nested
nature of the data and dependency of some effect sizes coming from the same
participants is acknowledged in the model.
Thank you!
Catia
--
C?tia Margarida Ferreira de Oliveira
Psychology PhD Student
Department of Psychology, Room A105
University of York, YO10 5DD
Twitter: @CatiaMOliveira
pronouns: she, her
[[alternative HTML version deleted]]
_______________________________________________ R-sig-meta-analysis mailing list R-sig-meta-analysis at r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
-- C?tia Margarida Ferreira de Oliveira Psychology PhD Student Department of Psychology, Room A105 University of York, YO10 5DD Twitter: @CatiaMOliveira pronouns: she, her