Skip to content

[R-meta] Question about Meta analysis

6 messages · Wolfgang Viechtbauer, Sevilay Cankaya, Maximilian Steininger

Original
Sevilay Cankaya
# 
Dear Steinininger,

I am writing to ask some questions about dependency in meta analysis. I
read your questions in the meta analysis group. I realised that ? have
similar questions. I am currently working on meta analysis about the
effectiveness of psychotherapies on juveniles psychology. I have 42 effect
sizes from 18 studies and the differences from your meta analysis is that ?
have multiple outcomes (depression, anger, mindfulness)  and ? want to
combine them as a psychosocial outcome. First I tried three level meta
analysis. Then while researching,  I saw that . clubsandwich, , RVE was
more suitable for my data. But I'm not sure because it is my first time.
 I want to ask how you deal with these issues(model selection).
And Do you have any resources or ideas that can help me with this?

Sincerely,
Sevilay ?ankaya
Wolfgang Viechtbauer
# 
Dear Sevilay,

I am not sure to whom you meant to write (you posted to the mailing list and I don't know who 'Steinininger' is), but you might find the following of relevance to your question:

https://wviechtb.github.io/metafor/reference/misc-recs.html#general-workflow-for-meta-analyses-involving-complex-dependency-structures

Best,
Wolfgang
Maximilian Steininger
# 
Dear Wolfgang, dear Selivay,

I think Selivay was referring to my longer message from a few days ago (see below). However, as I am only just starting to familiarise myself with the method, I am unfortunately unable to provide Selivay with any conclusive/helpful answers.

I had hoped that my open questions from back then might still be answered, but perhaps they are too obvious or uninformed (or simply too long) and can be answered with more literature research by myself.

Many thanks in any case for the link Wolfgang.

@Selivay: You can write me a direct message via maximilian.steininger at univie.ac.at , then I can share you a detailed list of all the resources I used.

Best,
Max
Wolfgang Viechtbauer
# 
Ah, now I get it. Then let me answer your other post here and maybe this will be of use to all.

As noted in my answer to Sevilay, this part of the metafor documentation is relevant:

https://wviechtb.github.io/metafor/reference/misc-recs.html#general-workflow-for-meta-analyses-involving-complex-dependency-structures

This is in essence your Q1, and yes, this is good practice. Not sure if this is 'best' practice. In general, how such complex cases should be handled depends on many factors.

Not sure what distinction you are making between this approach and the use of multivariate meta-analysis (combined with RVE), since the three-level model can also be seen as a multivariate meta-analysis, as discussed in these examples:

https://www.metafor-project.org/doku.php/analyses:konstantopoulos2011
https://www.metafor-project.org/doku.php/analyses:crede2010

A major challenge in cases where there is sampling error dependency is the construction of the V matrix. Many will not even attempt this and will rely on / hope that RVE fixes up the standard errors of the fixed effects. Roughly, this is at least asymptotically true as long as the cluster variable used in RVE encompasses estimates that are potentially dependent (due to whatever reason). In principle, the vcalc() function can handle quite a number of different types of dependencies for constructing the V matrix, but I even struggle at times trying to make it fit to a particular case. For example, this example shows this so some extent:

https://wviechtb.github.io/metadat/reference/dat.knapp2017.html

The other challenge is the choice of the random effects. Often, people just use a 'simple' three-level model, but more complex structures are certainly possible and may provide a better reflection of the depedency structure. An example where we did not use a V matrix (which would have been hopelessly complex) but used a more complex random effects structure is this:

https://wviechtb.github.io/metadat/reference/dat.mccurdy2020.html

With respect to your other questions:

Q2) Yes, I would say the test-retest reliability can be a decent proxy for estimating the correlation between estimates that are obtained at multiple time points (assuming that the time lags are similar).

Q3) As you note, the pre-post correlation is needed to correctly compute the sampling variance of a standardized mean change (with raw score standardization). That's a different issue than using a correlation coefficient to account for the dependency between two such effect sizes. So no, you are not being overly conservatively in doing so.

Q4) You do not need to 'correct' the control / common comparator group sample size when you account for the dependency via their covariance in the V matrix.

Q5) Hard to say without digging into the details of your data. But again, the three-level model *is* already a particular type of multivariate model. This aside, yes, these two ideas -- that there are multiple levels plus multiple types of outcomes -- can certainly be combined.

In general, I would say you are asking the right questions and are on the right track, but it is hard to say more without further details.

Best,
Wolfgang
Sevilay Cankaya
# 
Dear Wolfgang,

First of all I am sorry for my misunderstanding

Thank you for your advice, I will follow the link

Sincerely,
Sevilay ?ankaya

On Tue, 23 Apr 2024, 10:56 am Viechtbauer, Wolfgang (NP), <
wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:

            

      
      
      
    

        

  
  


  


      

    

    
2 days later

    

      
      

        


  

        

      Maximilian Steininger
    

    

      
      #
    

  


  

      
Dear Wolfgang,

Thank you very much for your detailed reply. I also wanted to take the opportunity to thank you for your extremely well-documented resources. They are enormously helpful.

Indeed, creating the V matrix is not trivial, but your examples are a great guide. If it's not too much to ask (if it is, nevermind!) I would appreciate feedback on my approach. Here is an exemplary structure from my data, which captures all of the dependencies that I have in my real dataset.

Study 1: Between-study with one intervention and one control group.
Study 2: Between-study with two interventions and one control group.
Study 3: Within-study with one control and one intervention condition (assumed test-retest r = 0.9).
Study 4: Within-study with one control and two intervention conditions (assumed test-retest r = 0.9).
Study 5: Between-study with two experiments, each containing two interventions and one control group.
Study 6: Within-study in which two different (low correlation, r= .1) dependent variables were used (assumed test-retest r = 0.9).

This is the (made-up) data set:

studyid = c(1,rep(2,2),3,rep(4,2),rep(5,4),rep(6,2))
esid = c(1:12)
design = c(rep(1,3), rep(2,3), rep(1,4), rep(2,2))
subgroup = c(rep(1,8), rep(2,2), rep(1,2))
type = c(rep(1,11),2)
time1 = rep(1,12)
time2 = c(rep(1,3), rep(2,2), 3, rep(1,4), rep(2,2))
grp1 = c("e","e1","e2","e","e","e","e1","e2","e1","e2","e","e")
grp2 = rep("c",12)
ne = c(10,15,20,30,40,40,45,50,80,100,90,90)
nc = c(11,16,16,30,40,40,46,46,61,61,90,90)
yi = seq(0.1, 1.2, by = 0.1)
vi = rep(0.05, 12)
dat = cbind.data.frame(studyid, esid, design, subgroup, type, time1, time2, grp1, grp2, ne, nc, yi, vi)

This would be my V matrix:

V = vcalc(vi=vi, cluster = studyid, subgroup = subgroup, type = type, time1 = time1, time2 = time2, grp1 = grp1, grp2 = grp2, w1 = nc, w2 = ne, rho = 0.1, phi = 0.9, data = dat)

And this is how I would specify the meta-analytic model:

res <- rma.mv(yi, V, random = ~ 1 | studyid/esid, data=dat)

What still puzzles me about the V matrix is why no dependence for study 5 between experiment 1 and experiment 2 is modeled (which might be unnecessary because this is taken care of by the random effect structure of three-level model?) and why the correlation of the effects in study 4 is 0.95 and not equal to my specification of phi = 0.9.

I also have a follow-up question regarding the use of SMD with raw score standardization. Calculating SMD for within-designs based on the raw-score metric, as suggested by Becker (1988), induces the problem that most within-studies use a counterbalanced design, and therefore there is no clear SDpre. Can this be ignored, or how should one best deal with it?

Thank you very much for your support. If it is too tedious to answer all my questions, please just ignore them.

Best,
Max