Skip to content

[R-meta] (Too) Many effect sizes for one single group

4 messages · Cátia Ferreira De Oliveira, James Pustejovsky

Cátia Ferreira De Oliveira
# 
Dear Wolfgang,

I hope you had a lovely start to the year.
I am sorry for starting the year with questions, but I just wanted to check
whether there is any drawback from including a lot of effect sizes from a
single paper when most labs contributed to the meta-analysis with just one
or two effect sizes? This resulted in a dataset where half of the effect
sizes come from multiple experiments run by the same group. The nested
nature of the data and dependency of some effect sizes coming from the same
participants is acknowledged in the model.

Thank you!

Catia
4 days later
James Pustejovsky
# 
Hi Catia,

Too add to the discussion, the data structure that you've described is
one where I think some additional caution is warranted. If half of the
effect size estimates in the synthesis come from a single paper (call
it Paper A), whereas most of the other studies contribute just one or
two effect sizes, then it makes me wonder whether there may be
something qualitatively distinctive about Paper A. Are all of the
effect sizes in Paper A equally comparable to the effect sizes from
the other papers? Or are the effect sizes in Paper A instead covering
sources of heterogeneity that are unexplored in the other papers?

An example might make my concern a little bit clearer. Say that you're
studying a phenomenon where there are several different scales for
operationalizing the outcome. Say that you've got 21 studies. In 20 of
them, the effect sizes are based one of two different outcome scales.
But in the 21st study (Paper A), the investigators measured both of
the commonly used outcome scales as well as 10 other scales that all
purport to measure the construct. So with this Paper A, the effect
sizes are heterogeneous in a way that we don't see in any of the other
studies.

I think there's a reasonable argument here that the meta-analysis
should be conducted by first discarding the uncommon outcome scales
from Paper A, and including only the two common scales. The scope of
generalization is then more limited because you're looking only at
those two scales. But generalization to other outcome scales seems
very tenuous because there's really only one study that provides
evidence about heterogeneity across outcomes.

James

On Wed, Jan 5, 2022 at 1:33 PM C?tia Ferreira De Oliveira
<cmfo500 at york.ac.uk> wrote:
Cátia Ferreira De Oliveira
# 
Dear James,

Just to clarify, would this still be the case even if this one paper
contained 4 experiments all on different participants? This paper would
still be the one mostly driving the nested and lack of independence in the
effect sizes.
Thank you! I may limit meta-regressions on only a very few number of
predictors then.

Best wishes,

Catia
On Mon, 10 Jan 2022 at 15:40, James Pustejovsky <jepusto at gmail.com> wrote:

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      James Pustejovsky
    

    

      
      #
    

  


  

      
Hi Catia,

I don't think there's a general answer here. It really depends on the
specific constructs you're studying and on the details of the studies
in your synthesis.

That said, I'll keep speculating: If the only difference between the
paper that contributes most of the effect size (what I was calling
Paper A) and the remaining papers is that Paper A includes multiple
replications, then I would probably be more comfortable including all
of the effect sizes from Paper A. But often in practice, when you've
got a multiple-study paper, the different experiments reported in the
paper are testing different variations of the intervention or
different tweaks to the protocol. If that's the case, it really
depends on whether those variations are also examined in some of the
non-Paper A effect sizes.

James

On Mon, Jan 10, 2022 at 10:41 AM C?tia Ferreira De Oliveira

        
<cmfo500 at york.ac.uk> wrote: