Skip to content

[R-meta] A rather general question study/effect size ratio

4 messages · Michael Dewey, Farzad Keyhan

Farzad Keyhan
# 
Hello All,

All else equal, would it be more desirable to have (A) 40 studies each
with two estimates of effect size or (B) 20 studies each with 4
estimates of effect size?

My intuition is that (A) is more desirable than (B).

Would it also be more desirable to have (C) 80 studies each with 1
estimate of effect size over (A) and (B)?

ps. By "desirable" I mean higher generalizability to the target population.

Thank you,
Fred
Michael Dewey
# 
Dear Farzad

If having more studies means that the treatment has been tried in a 
wider range of settings then my vote is for more studies rather than 
more effects per study. If the settings are homogeneous but the outcomes 
are varied then more outcomes might improve generalisability.

Michael
On 04/02/2022 16:50, Farzad Keyhan wrote:

  
    
  


  


      

    

    

      
      

        


  

        

      Farzad Keyhan
    

    

      
      #
    

  


  

      
Good point, Michael.

However, I was thinking of this more from the perspective of sampling.
The existence of multiple effects per study (scenarios A & B) may be
taken to indicate that the estimates have been sampled under a
multistage sampling plan from a larger population of studies (i.e.,
first, studies were randomly sampled, then, from within them, some
effects were randomly sampled, under a 3-level MLMA model).

The existence of a single effect per study (scenario C) may be taken
to indicate that the estimates have been sampled under a simple random
sampling plan from a larger population of studies (i.e., effects were
equally likely to be randomly sampled from any study in the population
of studies).

It would seem to me that generally the more the studies the better our
overall sense of the overall effect in the full population of studies.

Of the three scenarios (C) is the trickiest to me. It's a tough call
to say because each study has provided a single effect, then effects
were equally likely to be randomly sampled from any study in the
population of studies. Maybe if that had been the case, we would have
seen some studies with more than a single effect.

Perhaps all of this takes us back to using some form of
robust/sandwich estimation of effects, even if we have a single effect
per study.

Fred

        
On Fri, Feb 4, 2022 at 11:19 AM Michael Dewey <lists at dewey.myzen.co.uk> wrote:

            

      
      
      
    

        

      
      
    

            

      
      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Farzad Keyhan
    

    

      
      #
    

  


  

      
Oops, I forgot to add and ask this, the conceptual conflict in mind is that:

1- On the one hand, the more the studies vs. effects within studies,
the higher the generalizability of our meta-analytic results.

2- On the other hand, studies with more effects can influence the
studies with a single or fewer effects.

It seems to me 1- and 2- are at odds with each other meaning that:

A collection of studies each with a single effect gives us the most
precise meta-analytic estimates (i.e., rma() type model).

Yet, the existence of single-effect studies in a group of
multiple-effect studies reduces the precision of our meta-analytic
estimates such that multiple-effect studies should influence
single-effect studies to improve the precision of the meta-analytic
estimates (i.e., rma.mv() type model)

        
On Fri, Feb 4, 2022 at 12:10 PM Farzad Keyhan <f.keyhaniha at gmail.com> wrote: