Skip to content

[R-meta] aggregating effect sizes

6 messages · Filippo Gambarota, Wolfgang Viechtbauer, James Pustejovsky

Filippo Gambarota
# 
Hi,
In order to be sure which function to use I would like to ask if the
aggregation method of multiple effect sizes with dependent sampling
error suggested by Borenstein et al. (2009) is the same as what
performed by the aggregate() function in metafor specifying a single
correlation.
In my case I have calculated pre-post effect size using Morris (2008)
and then I have to combine multiple effect sizes calculated on the
same pool of subjects.
Thank you!
Wolfgang Viechtbauer
# 
Dear Filippo,

If you are asking about what is described in Box 24.1, then the answer is yes, if you use struct='CS' (which is the default) and 'weighted=FALSE' -- the default in aggregate() is to compute a weighted average, but Borenstein et al. only give the equations for computing an unweighted average and its sampling variance (but since the sampling variances of the two estimates that are being aggregated in the book example are the same, whether one uses weighted=TRUE or FALSE makes no difference). You can also find the corresponding code here:

https://wviechtb.github.io/meta_analysis_books/borenstein2009.html#24)_Multiple_Outcomes__Time-Points

Best,
Wolfgang
Filippo Gambarota
# 
Thank you Wolfgang,
So if I get correctly, the weighted approach should be preferred if values
that I have to aggregate are quite different? Because using the Borenstein
vs weighted method gives me quite different results, especially for the
mean effect.
Thank you!

On Mon, 10 Jan 2022 at 12:57, Viechtbauer, Wolfgang (SP) <
wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:

            

      
      
      
    

        

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Wolfgang Viechtbauer
    

    

      
      #
    

  


  

      
When you say the 'values', do you mean the estimates themselves or their sampling variances?

I only talked about the latter, that is, if the sampling variances within a set of estimates are homoscedastic, then there is no difference between weighting or not weighting. If the sampling variances are quite different, then I would typically prefer to use weighting, since that will give the most efficient estimate of the underlying true effect/outcome for the set.

However, James Pustejovsky (cc-ed) asked me to add the 'weighted' option to aggregate(), because there can be circumstances where using a simple (unweighted) average might be preferred.

If I recall, one argument goes along the following lines. Say you want to aggregate two effect estimates, one for male and one for female subjects. With weighting, the two estimates are weighted approximately proportional to the sample sizes within the two subgroups. However, the subgroup sizes within a study are just a reflection of how many male and female subjects the researchers were able to recruit for their study (and females tend to be more likely to volunteer), which doesn't reflect the population to which you want to make an inference (which consists of approx. equal parts of male and female subjects). So in that case, a simple average might be preferred.

Best,
Wolfgang

            

      
      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Filippo Gambarota
    

    

      
      #
    

  


  

      
Thanks, this is quite helpful. I've noticed that in my case,
aggregating with and without weighting changes the estimated effect
size noticeably. As you said this is correct because we are doing an
inverse-variance weighted average between effect sizes and I was wondering
which is the best approach. So aggregating with weighting could be
considered as a fixed-effect model that takes into account the correlation.

On Mon, 10 Jan 2022 at 16:09, Viechtbauer, Wolfgang (SP) <

        
wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:

        

            

      
      
      
    

        

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      James Pustejovsky
    

    

      
      #
    

  


  

      
Wolfgang's example about aggregating across two subgroups is a good
one. I'll add another, more abstract concern. If you are assuming a
common correlation, and if the sampling variances of your effect size
estimates are unequal, then inverse-variance (really
inverse-variance-covariance) weighting can assign *negative* weight to
effect sizes with larger sampling variances. Weird things can happen
with negative weights, such as getting aggregated effect sizes that
are larger than the maximum of all the component effect sizes (or
smaller than the minimum of all the component effect sizes). So it's
something to look out for.

To be more specific, let's say you're assuming a correlation of r and
that study j contributes k_j effect size estimates, with standard
errors s_1j, s_2j,...., etc. Using inverse-variance weighting, study i
will get negative weight if
1 / s_ij < (r / [ r (k_j - 1) + 1]) * sum_{h = 1}^{k_j} 1 / s_hj
That is, if 1 / s_ij is less than the fraction (r / [ r (k_j - 1) +
1]) of the total of the inverse standard errors. With r = 0.7 and k_j
= 3, this fraction is 0.7 / 2.4 = 0.29. So if one of the effect sizes
has 1 / s_ij that is less than 29% of the total of those terms, it
will get negative weight. Thus, this can happen even if the
discrepancy between sampling variances is pretty small.

James

On Mon, Jan 10, 2022 at 9:09 AM Viechtbauer, Wolfgang (SP)

        
<wolfgang.viechtbauer at maastrichtuniversity.nl> wrote: