[R-meta] imputing covariance matrices for meta-analysis of dependent effects

Van den Noortgate, W., L?pez-L?pez, J. A., Mar?n-Mart?nez, F., &
S?nchez-Meca, J. (2013). Three-level meta-analysis of dependent effect
sizes. Behavior Research Methods, 45(2), 576?594.
https://doi.org/10.3758/s13428-012-0261-6

Hi James,

This is indeed useful.

However, I am not sure if you are computing the covariances correctly. You
are essentially using:

covariance = correlation * sqrt(variance1 * variance2)

But apparently, if one goes through the derivation for the covariance for
standardized mean differences, one ends up with a different equation
(equation 10 in Kalaian and Raudenbush, 1996). In fact, the equation for
the covariance depends on the measure used. See, for example:

Wei, Y., & Higgins, J. P. (2013). Estimating within-study covariances in
multivariate meta-analysis with multiple outcomes. Statistics in Medicine,
32(7), 1191-1205.

Historical side note: Interestingly, the 'covariance = correlation *
sqrt(variance1 * variance2)' equation for standardized mean differences was
also used in:

Raudenbush, S. W., Becker, B. J., & Kalaian, H. (1988). Modeling
multivariate effect sizes. Psychological Bulletin, 103, 111-120.

but this was later corrected in Kalaian and Raudenbush (1996) based on
Gleser and Olkin (1994).

In practice, it probably makes relatively difference how exactly one
computes those covariances, esp. if one is 'guestimating' the correlation
between the measures anyway (and then follows things up with some kind of
cluster-robust approach, as you describe on your blog). As far as I am
concerned, it is important though that one actually computes some kind of
covariances (to get a better 'working' var-cov matrix to begin with). I am
seeing an increasing number of papers where multiple effect size estimates
based on the same sample (so, multivariate data) are being analyzed using a
multilevel model like the one described by Konstantopoulos (2011). But that
model assumes that sampling errors are uncorrelated, so this is a
misapplication of that model. That is also why I added this at one point:

http://www.metafor-project.org/doku.php/analyses:konstantopoulos2011#
uncorrelated_sampling_errors

As for the different results reported in Kalaian and Raudenbush (1996) and
the ones obtained with metafor -- a couple years ago, I also entered those
data (from Table 1) and tried to re-analyze the dataset (as I was planning
to include that dataset also in metafor) and ran into similar
discrepancies. In fact, if one re-creates the scatterplots that are shown
in Figure 1 from the data reported in Table 1, it becomes clear that there
must be several printing errors. So, it's not really possible to reproduce
the results from that paper, which is unfortunate, since it would be a nice
illustration of the multivariate approach.

Best,
Wolfgang

--
Wolfgang Viechtbauer, Ph.D., Statistician | Department of Psychiatry and
Neuropsychology | Maastricht University | P.O. Box 616 (VIJV1) | 6200 MD
Maastricht, The Netherlands | +31 (43) 388-4170 | http://www.wvbauer.com

-----Original Message-----
From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-
bounces at r-project.org] On Behalf Of James Pustejovsky
Sent: Thursday, August 10, 2017 17:59
To: Michael Dewey
Cc: r-sig-meta-analysis at r-project.org
Subject: Re: [R-meta] imputing covariance matrices for meta-analysis of
dependent effects

Michael,

I was not aware of the metavcov package, so thank you for pointing it out.

On first glance, it looks like the metavcov functions are configured based
on the assumption that you have detailed information about the correlations
between outcomes for each study (i.e., it requires a list of correlation
matrices as input). The function from my previous message is a simpler
utility function, for use when you need to make more or less ad hoc
assumptions about the correlations. So I would say that it does complement
the metavcov package, but I would welcome corrections if this is not an
accurate assessment.

Best,
James

On Thu, Aug 10, 2017 at 10:43 AM, Michael Dewey <lists at dewey.myzen.co.uk>
wrote:

Dear James

Not sure how relevant this is but does it complement in any way the
package  https://CRAN.R-project.org/package=metavcov ? I have not used

it

by the way.

Michael

On 10/08/2017 15:04, James Pustejovsky wrote:

All,

A common problem in multivariate meta-analysis is that the information
needed to calculate the correlation between effect size estimates is not
reported in available sources, even when the variances of the estimates
can
be calculated. One approach to handling this situation is to simply make
an
informed guess about the correlation between the effect sizes. I use

this

approach fairly often and have written a function that makes some of the
calculations easier. The function calculates a block-diagonal
variance-covariance matrix based on the sampling variances and a guess
about the degree of correlation. More details available here:

http://jepusto.github.io/imputing-covariance-matrices-for-
multi-variate-meta-analysis

There's nothing innovative about the methods I describe, but I figured
that
others might find the function useful. I would welcome comments,
questions,
or debate about the utility of the approach I used.

Cheers,
James