Skip to content

[R-meta] correct tau interpretation three-level meta-analysis

4 messages · Filippo Gambarota, Wolfgang Viechtbauer, James Pustejovsky

Original
Filippo Gambarota
# 
Hi,
I'm conducting a meta-analysis using a three-level model because I
have multiple effects within the same study. My model is something
like this:

```
rma.mv(yi, vi, random = ~1|study/outcome)
```
My question is simply how to correctly interpret tau at the outcome
level. The tau at the study level is basically the standard
heterogeneity. The tau at the outcome level is the average variability
within a cluster (study)?
Thanks
1 day later
Wolfgang Viechtbauer
# 
Dear Filippo,

The variance component corresponding to 'study' is the amount of heterogeneity in the average true effects of the various studies (i.e., studies with multiple outcomes have an average true effect). The variance component corresponding 'outcome' is the amount of heterogeneity in the true effects around those average true effects of the studies.

So, I would say that neither is really the standard heterogeneity, but that also depends on what exactly you mean by that.

Best,
Wolfgang
Filippo Gambarota
# 
Thank you!
Sorry I used the word standard badly. With standard, I mean the usual
between-study heterogeneity that we estimate in a random-effect model.
Given that the `outcome` heterogeneity is the variability within each study
cluster (i.e. the variability around the true effect for each `study`), my
question is:
Do we interpret it as an average variability within each cluster among
clusters? Or we are assuming that each cluster has the same within-cluster
variability?
Thank you!

On Mon, 5 Jul 2021 at 17:04, Viechtbauer, Wolfgang (SP) <
wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:

            

      
      
      
    

        

  
    

      
      
    

  


  


      

    

    
1 day later

    

      
      

        


  

        

      James Pustejovsky
    

    

      
      #
    

  


  

      
Hi Filippo,

To add to Wolfgang's response, one note of caution regarding
interpreting the variance components in the three-level meta-analysis
(3LMA) model is that the variance component estimates are somewhat
sensitive to assumptions. If your data structure involves multiple,
correlated effect size estimates (i.e., estimates based on the same sample
of participants, so that the sampling errors of the estimates are
correlated), then the 3LMA model involves some degree of model
mis-specification. Currently available evidence suggests that the 3LMA may
be fairly robust with respect to inferences about *average effect
sizes*---that is, even though the model is mis-specified, hypothesis tests
and confidence intervals based on the model still have calibration rates
that are close to correct.

This robustness property does NOT extend to estimation of variance
components. If the model is mis-specified, then there will generally be
some degree of systematic bias in the variance component estimates. For
instance, say that the true correlation between effect size estimates from
the same sample is around r = 0.6. Using the 3LMA is equivalent to assuming
r = 0.0. As far as I understand, this will lead to estimates of
within-study heterogeneity that are systematically *too small* and
estimates of between-study heterogeneity that are systematically *too
large*. How strong the biases are depends on the structure of your data, so
it's hard to say much further here.

To your other question:

            

      
      
      
    

        
I would say that the answer is "both." As formulated, the 3LMA model does
make the assumption that each cluster has the same within-cluster variance
component (i.e., homogeneity of variance within clusters). But even if this
assumption is incorrect, the estimated within-cluster variance will be some
sort of weighted average of the within-cluster variances, at least at an
approximate level. In principle, you could estimate cluster-specific
variances using the following (assuming that every value of outcome is
unique across studies):
```
rma.mv(yi, vi, random = list (~1|study, ~ study | outcome, struct = "DIAG")
```
But this probably isn't a good idea unless you have a lot of estimates from
every cluster. And the comments above regarding model mis-specification
apply here as well.

Kind Regards,
James