Looks good.
A few points:
About the warning: "Some combinations of the levels of the inner factor never occurred. Corresponding rho value(s) fixed to 0." Indeed, this is a result of some pairs of outcomes never occurring within studies. Hence, it is not possible to estimate the correlation for such pairs.
With 24 different outcome measures for 28 effect sizes, struct="CS" is indeed the most you can do. Any more complex structure would either be a total overfit or will not converge. Note that struct="CS" implies that the amount of heterogeneity is the same for all outcomes and that the correlation between outcomes is the same, regardless of which pair of outcomes you are looking at. Obviously, the results from your model are then an approximation to a more complex reality.
In most cases, whether you use metafor or clubSandwich for the 'cluster robust' computations won't make much of a difference. The nice thing about clubSandwich is that it provides some refined robust methods (esp. vcov="CR2" and vcov="CR3") that should in principle lead to more accurate results in small samples. You might as well use them.
Note that cooks.distance() won't work with an object returned by robust(). So, you would have to directly apply cooks.distance() to 'MultiMeta'. But that should still be okay as a general diagnostic tool.
I would suggest to present the results with and without the outlier.
Also, you cannot get a test for (residual) heterogeneity based on (cluster) robust methods. A test for (residual) heterogeneity is based on a model that assumes that the only source of variability (and covariation) is contained/captured by V. But (cluster) robust methods try to approximate all source of variability (and covariation), not just those due to V. Given that V in your case is just a rough approximation, any test for (residual) heterogeneity should also be treated that way.
Best,
Wolfgang
-----Original Message-----
From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces at r-project.org] On Behalf Of schlegei
Sent: Thursday, August 17, 2017 15:23
To: r-sig-meta-analysis at r-project.org
Subject: Re: [R-meta] Multivariate meta-analysis with unknown covariances?
First of all: Thanks a lot to the two of you for your kind and very
helpful answers!
What a fortunate coincidence that James Pustejovsky published his blog
entry the same day I was asking how to deal with unknown covariances.
Does anyone know a published reference, in which the three steps are
recommended? Until now, I couldn?t find one.
I want to share my R-Code with the list (some explanations included).
Maybe someone more experienced might check if my specification is
correct. And maybe it helps other clueless people with the same problem:
1. Calculation of the effect sizes:
data <- escalc(measure = "SMD", m1i = m12, sd1i = sd12, n1i = n12, m2i =
m22, sd2i = sd22, n2i = n22, data = data, append = TRUE, replace =
FALSE)
data <- escalc(measure = "PBIT", ai = a1, bi = b1, ci = c1, di = d1,
data = data, append = TRUE, replace = FALSE)
I calculated the effect sizes with Hedges g (?SMD?) and transformed
dichotomized data to standardized mean differences with the help of the
probit transformed risk difference (?PBIT?).
2. Imputation of the variance-covariance matrix:
Vlist <- impute_covariance_matrix(vi = data$vi, cluster = data$study, r
= 0.7)
Right now, I fixed the correlation between all outcomes in the same
study to 0.7. This is quite rough and I want to precise this guestimates
(I asked most of the original autors if they can provide me with the
correlations between the outcomes and will also precise this guestimate
by substituting it with correlations from other studies that used the
same outcomes).
3. Conduct the multivariate meta-analysis:
MultiMeta <- rma.mv(yi = yi, V = Vlist, mods = ~ factor(controlgroup)
-1, random = ~ factor(outcome)|study, struct = "CS", data = data)
I formulated a multivariate meta-analysis with random effects and
included the imputed covariance matrix into the model. I had to fix the
structure to a compound symmetric structure (?CS?), because with less
restrictive structures I received the following warning message:
Fehler in rma.mv(yi = yi, V = VPostdicho, mods = ~factor(Kontrollgruppe)
- : Optimizer (nlminb) did not achieve convergence (convergence = 1).
Zus?tzlich: Warnmeldung: In .process.G.afterrmna(mf.g, g.nlevels,
g.levels, struct[1], tau2, :Some combinations of the levels of the inner
factor never occurred. Corresponding rho value(s) fixed to 0.
Probably the warning message is due to the fact that I have 24 different
outcome measures for 28 effect sizes ? so there is no combination of
outcomes that appears several times. In addition to that, I included a
categorical moderator (mods = ~ factor(controlgroup) -1) in the
meta-analysis. When I set e.g. struct = ?UN?, the optimizer is not
converging. With struct = ?CS?, it is.
So setting struct = ?CS? seems to be the only possiblity here?
4. Compute robust tests and confidence intervals:
I tried both options (in metafor as well as in club sandwich) to
estimate robust standard errors and p-values:
metafor_robust <- robust.rma.mv(MultiMeta, cluster = data$study)
ClubSandwich_robust <- coef_test(MultiMeta, vcov = "CR2")
Both options result in similar (but not identical) values. Probably, for
my research it is interchangeable which option I choose?
For further analysis (like Cook?s distance or Egger?s test) one uses
exclusively the robust estimates, or?
In my case, the test for residual heterogenity in the model with the
imputed covariance matrix is highly significant. When I exclude one
effect size that is an outlier, the residual heterogenity is not
significant anymore. May I present this result, although it refers to
the model with the imputed covariance matrix? (There is no test for
residual heterogenity for robust estimates)
Best regards,
Isabel Schlegel
Am 10.08.2017 21:23, schrieb Viechtbauer Wolfgang (SP):
Indeed, unknown correlations seems to be a 'hot topic' right now.
Let me first clarify that metafor cannot somehow magically solve the
problem with missing covariances. One has to do some extra work to
deal with this issue. The post that Isabel refers to
(https://stat.ethz.ch/pipermail/r-sig-mixed-models/2015q2/023727.html)
discusses some possibilities. A defensible strategy is to:
1) Guestimate the unknown correlations and then compute approximate
covariances between the effect size estimates.
2) Fit a proper multivariate model.
3) Follow things up with a cluster-robust approach.
This is basically what James just described in his post and on his
blog (and what Emily refers to):
http://jepusto.github.io/imputing-covariance-matrices-for-multi-variate-meta-analysis
And so, yes, I think you can conduct a multivariate meta-analysis
based on the data set described.
Best,
Wolfgang
-----Original Message-----
From: R-sig-meta-analysis
[mailto:r-sig-meta-analysis-bounces at r-project.org] On Behalf Of Emily
Finne
Sent: Thursday, August 10, 2017 20:25
To: r-sig-meta-analysis at r-project.org
Subject: Re: [R-meta] Multivariate meta-analysis with unknown
covariances?
Dear Isabel,
I'm not an expert in meta-analysis, so I have to leave your question -
if multivariate MA makes sense at all in your case - open to the
experts
around. If your different outcomes do measure different symptoms
originating in different disorders I would have my doubts if it makes
sense to combine them.
But it seems to me that the follwing recent post is addressing your
problem with the missing covariances quite well:
https://stat.ethz.ch/pipermail/r-sig-meta-analysis/2017-August/000094.html
How to construct the var-cov matrix for multiple endpoint studies in
metafor is illustrated here:
http://www.metafor-project.org/doku.php/analyses:gleser2009. (But you
would have to guess a within-study correlation between different
outcomes).
Good luck with your thesis!
Best,
Emily
Am 10.08.2017 um 18:54 schrieb schlegei:
for my master thesis, I want to conduct a multivariate meta-analysis
with the R-package metafor.
Unfortunately, I?m not sure if it is possible to conduct this analysis
based on my data set.
To illustrate my problem, a few words about my research question: I
investigate the efficacy of gestalt therapy (a psychotherapeutic
approach) for people with a mental disorder according to DSM-IV/ICD-10
(my focus is on symptom reduction). My literature research resulted in
12 randomized controlled trials (RCTs).
From this 12 studies, I extracted 28 outcomes and calculated the
effect sizes and variances (standardized mean differences). I assume
outcomes from the same study are dependent. Unfortunately, in no study
correlations/covariances between the sampling errors of the outcomes
are reported. So the within study covariance structure is totally
missing.
Because I included studies about people with different mental
disorders, my outcomes are pretty heterogeneous: Only one
questionnaire (Beck?s Depression Inventory) was used in 4 studies,
apart from that the outcomes don?t overlap between the studies.
Summed up, my data set consists of the following variables:
ES_ID = idenfification number for every effect size (I have 28 effect
sizes)
study = every study gets one number (I included 12 studies)
outcome = every questionnaire/outcome gets one letter (I included 24
different outcomes)
yi = effect size
vi = variance of the effect size
Is it possible to conduct a multivariate meta-analysis based on this
data set?
My supervisor told me, the missing within study covariances can be
easily estimated with the R-package metafor. Up until now, I do not
understand how. Following the discussions on stackexchange and this
mailing list (e.g.
https://stat.ethz.ch/pipermail/r-sig-mixed-models/2015q2/023727.html),
it seems to me that estimating the whole covariance structure is not
that easy and is attended by some disadvantages/assumptions . This
also coincides with other articles I have read about multivariate
meta-analysis and in which missing covariances are described as a
major problem. When I contacted my supervisor, he just told me that
the literature I read is out-dated and repeated that the problem of
missing covariances can be solved with metafor (unfortunately, he
didn?t recommend up-to-date articles to me).
Right now, I feel a bit locked in a stalemate. Is there a simple,
up-to-date solution for my problem with the missing covariances that I
have overseen?
If yes: I would be extremely happy about any tip!
If no: Do you think, it makes sense to conduct a multivariate
meta-analysis based on my data set or is it more appropriate to choose
one effect size per study (univariate meta-analysis)? If it is
possible to conduct a multivariate meta-analysis based on my data: Is
there a strategy ? like making a rough guess of the correlations or
using robust methods ? that you would recommend?
I would be really happy to hear a response,
Isabel Schlegel