Skip to content

[R-meta] Coding multi-measure correlational studies for multilevel meta-analysis

4 messages · Yuhang Hu, Wolfgang Viechtbauer

Yuhang Hu
# 
Hello Experts,

I'm collecting the correlations between 8 variables from several studies.
If a study has used a single measure for all these 8 variables, I will need
28 rows (assuming no missing) to capture all those correlations i.e.,
var1.var2 = combn(1:8, 2, FUN=\(i)paste(i,collapse = ".")):

    study   ri   var1.var2
1)  1        .1   1.2
 ...
28) 1       .2    7.8

But if a study has used, say, two measures (e.g., 1, 2) for two of those 8
variables (e.g., variables "1" and "2" in 'var1.var2'), then, I wonder how
**best** to capture the additional 13 correlations arising due to the
additional measure used for "1" and "2" in that study in my data for
multilevel modeling purposes?

One approach might be to add a single column called, say "measure" to add
just those additional rows in that multi-measure study:

    study   ri   var1.var2  measure
1)   1       .1    1.2
 ...
6)   1       .6     1.7           1
7)   1       .4     1.7           2
...
12)  1       .8     2.7          1
13)  1       .7     2.7          2
...

But this looks messy. For instance, what should be the value of "measure"
for the var1.var2 rows that have used a single measure (e.g., var1.var2 ==
1.2)? And can "measure" coded this way be used in the random part of the
model (metafor::rma.mv)?

Thanks,
Yuhang
Wolfgang Viechtbauer
# 
Hi Yuhang,

First of all, I would suggest to create two separate variables for the two variables, like this:

    study    ri   ni  var1  var2
1)  1        .1   85  1     2
...
28) 1        .2   85  7     8

Then you can use rcalc() to create the var-cov matrix of the (raw or r-to-z transformed) correlation coefficients within studies (the 'V' matrix), that is, if your dataset is called 'dat', you can do:

tmp <- rcalc(ri ~ var1 + var2 | study, ni=ni, data=dat)
V <- tmp$V
dat <- tmp$dat

Sidenote: For 8 variables, there are 8*7/2 correlations (or generally, for p variables, p*(p-1)/2 -- this is one of those equations one eventually has memorized due to using it so often).

For a study (say, study 2) that used multiple measures for one of the variables (say, variable 8), there are then actually 9 variables and hence 9*8/2 = 36 correlations. The structure then is:

    study    ri   ni  var1  var2  measure1 measure2
1)  2        .1   78  1     2     a        b
...
7)  2        .3   78  1     8     a        x
8)  2        .2   78  1     8     a        y
9)  2        .4   78  2     3     b        c
...
14) 2        .3   78  2     8     b        x
15) 2        .2   78  2     8     b        y
16) 2        .5   78  3     4     c        d
...
20) 2        .4   78  3     8     c        x
21) 2        .5   78  3     8     c        y
22) 2        .1   78  4     5     d        e
...
25) 2        .0   78  4     8     d        x
26) 2        .1   78  4     8     d        y
27) 2        .3   78  5     6     e        f
...
29) 2        .3   78  5     8     e        x
30) 2        .2   78  5     8     e        y
31) 2        .2   78  6     7     f        g
32) 2        .3   78  6     8     f        x
33) 2        .1   78  6     8     f        y
34) 2        .1   78  7     8     g        x
35) 2        .2   78  7     8     g        y
36) 2        .3   78  8     8     x        y

The actual values used for measure1 and measure2 are irrelevant, as long as you use them consistently within a study. For studies that only used a single measure for each variable, you can leave measure1 and measure2 blank. For studies that used multiple measures for more than one variable, you have to keep expanding this structure. It just becomes very tedious to construct.

Then for rcalc(), you need to paste together var1 and measure1 and var2 and measure2:

dat$v1m1 <- paste0(dat$var1, ".", dat$measure1)
dat$v2m2 <- paste0(dat$var2, ".", dat$measure2)

and use those in rcalc():

tmp <- rcalc(ri ~ v1m1 + v2m2 | study, ni=ni, data=dat)
V <- tmp$V
dat <- tmp$dat

For the actual model fitted with rma.mv(), you don't use the combination of v1m1 and v2m2, but the combination of var1 and var2 as the predictor:

dat$var1var2 <- paste0(dat$var1, ".", dat$var2)

since you *want* the model not to give you estimates of 'measure-specific' pooled correlations, but you want to average over multiple measures for the same variable. So the model could be:

rma.mv(yi, V, mods = ~ 0 + var1var2, random = ~ var1var2 | study, struct="UN", data=dat)

However, this model will need to estimate 28*27/2 = 378 correlations plus 28 variances (tau^2 values) for the random effects, so in total 406 (!!) parameters (the general equation is p*(p+1)/2), plus the 28 fixed effects. That's a lot of parameters in the unstructured var-cov matrix of the random effects, so unless you have a lot of studies (hundreds if not thousands), this is going to be difficult or essentially impossible. This aside, the model allows for no heterogeneity when there are multiple correlations for the same var1var2 pair. A simple way to allow for this is to add another estimate specific random effect to the model:

dat$id <- 1:nrow(dat)
rma.mv(yi, V, mods = ~ 0 + var1var2, random = list(~ var1var2 | study, ~ 1 | id), struct="UN", data=dat)

This is simplistic, since it assumes that the heterogeneity in multiple correlations for the same pair is the same regardless of the pair. If you have a lot of data, one could try:

rma.mv(yi, V, mods = ~ 0 + var1var2, random = list(~ var1var2 | study, ~ var1var2 | id), struct=c("UN","DIAG"), data=dat)

which would use separate estimate-level random effects for each pair, but this adds another 28 parameters to the model. But who cares about another 28 if one already has 406 ...

Realistically, one needs to simplify the random effects structure. On the opposite end, there is the minimalistic:

res <- rma.mv(yi, V, mods = ~ 0 + var1var2, random = ~ 1 | study/id, data=dat)
res

which, due to its overly simplistic nature, really needs to be followed-up with:

robust(res, cluster=study, clubSandwich=TRUE)

(could do the same with the models above, but this is less likely to matter if one actually manages to fit these complex models).

An interesting question is what kind of structures of intermediate complexity one could consider.

But I'll stop here for now, since this is getting way too long anyway.

Best,
Wolfgang
Yuhang Hu
# 
Dear Wolfgang, thank you so much. A few observations.

1- This is, as you said, "very tedious to construct". So, I really wonder
if we "*want* the model not to give us estimates of 'measure-specific'
pooled correlations", then, can't we just average (maybe using
metafor::aggregate.escalc) across "ri" for different measures manually and
this way, reduce the data rows for a multi-measure study to 28 rows just
like a single-measure?

2- The difficulty of coding these studies extends to other
variable-specific moderators as well. For example, if I want to code for
the reliability of the variables in each pair, there again, things get
messy in multi-measure studies. So, here I should average over reliability
values for each variable across different measures?

3- What if the variable-specific moderators in a multi-measure study were
categorical? Say, qualitative features of the measures used (e.g., standard
vs. researcher-developed). Now, we can't average over this feature for each
variable across different measures. So what can we do?

4- Regarding rcalc(), I actually intentionally didn't use it, because, at
times, studies used multiple samples and times of measurement.

Thank you,
Yuhang

On Wed, Dec 13, 2023 at 7:06?AM Viechtbauer, Wolfgang (NP) <
wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:

            

      
      
      
    

        

  
  


  


      

    

    

      
      

        


  

        

      Wolfgang Viechtbauer
    

    

      
      #
    

  


  

      
Please see my responses below.

Best,
Wolfgang

            

      
      
      
    

        
In order to do this correctly, one would have to compute the var-cov matrix of the correlations one wants to pool over anyway. Of course, nobody is going to stop you from just averaging them and pretending that it is just a single correlation coefficient. It's just objectively wrong.

            

      
      
      
    

        
As above.

            

      
      
      
    

        
This would in fact be an argument for not averaging and include the various correlations corresponding to different measures (with a measure-specific moderator variable).

            

      
      
      
    

        
But rcalc() can handle this. Multiple samples are independent, so those just need to be treated (for the purposes of rcalc()) as different studies. And there is, from a computational point, no difference between multiple measures and multiple time points. It just creates more variables. Of course this complicates the construction of the dataset used as input to rcalc(), but that's a practical issue.

I totally get that constructing such a dataset is quite difficult and time-consuming, if not impossible. What I describe below is the ideal approach where one constructs the dataset for a study that includes every possible pair of variables, where variables can reflect different constructs, multiple measures of the same construct, and/or multiple timepoints. If this is not possible due to logistic/practical reasons, then one would have to consider alternative approaches. A rough var-cov matrix could still be constructed with the vcalc() function. One could even go as far as pretending V is diagonal. In any case, cluster-robust inference methods should then be used.