[R-meta] query re: multi-level analysis in metafor with missing standard errors - p.s.

Dear Jenny,

To clarify one potential misconception: The type of meta-analytic model
has no bearing on the way the standard error for an effect size measure
should be computed. So it is not relevant whether you will conduct a
multilevel meta-analysis or not.

Proper methods for computing the standard error of a standardized
regression coefficient are discussed in these articles:

Yuan, K.-H., & Chan, W. (2011). Biases and standard errors of standardized
regression coefficients. Psychometrika, 76(4), 670-690.

Jones, J. A., & Waller, N. G. (2013). Computing confidence intervals for
standardized regression coefficients. Psychological Methods, 18(4), 435-453.

Jones, J. A., & Waller, N. G. (2015). The normal-theory and asymptotic
distribution-free (adf) covariance matrix of standardized regression
coefficients: Theoretical extensions and finite sample behavior.
Psychometrika, 80(2), 365-378.

The results from Yuan and Chan (2011) are interesting, but not really
usable in practice because you would essentially need access to the raw
data to use their equations. Jones and Waller (2015) show how to do the
same thing, but using only correlations. You can do the same thing with
some functions that were recently added to metafor.

For the following to work, you need to install the 'devel' version of
metafor as described here:
https://github.com/wviechtb/metafor#installation

###################

library(metafor)

# unstandardized regression coefficients
res <- lm(mpg ~ cyl + disp + hp, data=mtcars)
summary(res)

# standardized regression coefficients
res <- lm(scale(mpg) ~ 0 + scale(cyl) + scale(disp) + scale(hp),
data=mtcars)
summary(res)

# NOTE: The SEs given above are only appropriate if we assume that the
null hypothesis is true and therefore cannot be used for a meta-analysis
(even if they were reported)

# correlation matrix
R <- with(mtcars, cor(cbind(mpg, cyl, disp, hp)))
R

# show that we can get the same results as above just based on R (and n)
matreg(1, 2:4, R=R, n=nrow(mtcars))

# the SEs are still the same as before and again should not be used for a
meta-analysis

# to compute SEs of the standardized regression coefficients that are
appropriate to use for a meta-analysis, we first need to compute the
(asymptotic) var-cov matrix of the correlation coefficients
sav <- rcalc(R, ni=nrow(mtcars))

# then we can use this to get SEs of the standardized regression
coefficients
matreg(1, 2:4, R=R, V=sav$V)

###################

I think the 'fungible' package also has a function for this and may also
provide the asymptotically distribution free method as described by Jones
and Waller (2015). In any case, even with this approach, you need the full
correlation matrix (of all predictors and the outcome). Maybe this is
available from the articles (or the authors).

Best,
Wolfgang

-----Original Message-----
From: R-sig-meta-analysis [mailto:

r-sig-meta-analysis-bounces at r-project.org]

On Behalf Of Jennifer Oser
Sent: Monday, 05 October, 2020 11:05
To: r-sig-meta-analysis at r-project.org
Subject: [R-meta] query re: multi-level analysis in metafor with missing
standard errors - p.s.

Hi all, I write with a p.s. to my post from last week (Sep 29) copied

below

that my co-authors and I are still trying to resolve:

In our dataset, 83 of the total 185 coded effects are in this problematic
category for multi-level analysis in metafor, i.e., a coefficient reported
in a study's regression table as a standardized coefficient with no
reported standard error.

My guess is that this must be a common issue for applied researchers like
myself and my co-authors who are interested in using this approach, as our
dataset includes a nice distribution of studies according to standard
indicators of article quality (we coded ISI impact factor quartile).

As noted in my original post, we are very keen to try to resolve this asap
as the rest of the manuscript is in final draft form and we've received
very favorable comments on the substantive academic contribution.

Any and all tips and hints are welcome!
Jenny

Dr. Jennifer Oser
Department of Politics & Government
Ben-Gurion University of the Negev
https://www.jenniferoser.com/

****************************************************************
Jennifer Oser o at er @end|ng |rom po at t@bgu@@c@||
Tue Sep 29 18:13:17 CEST 2020

Hi all,

My co-authors and I are conducting a multi-level meta-analysis, and I

write

with a question about calculating the sampling variance to use in the

rma.mv

argument. We followed the approach of calculating the sampling variance

(v)

by squaring the standard error, as documented in "Doing Meta-Analysis in

R"

here:

https://bookdown.org/MathiasHarrer/Doing_Meta_Analysis_in_R/fitting-a-three-

level-model.html
.
However, most of the regressions in the studies that meet our inclusion
criteria report on standardized coefficients without reporting standard
errors, and therefore cannot be included in the multi-level analysis using
this approach.

Our question is therefore: Is there a way to conduct a multi-level

analysis

that calculates the relevant sampling variance based on regression outputs
that report standardized coefficients but do not report standard errors?

We

know that it is possible to do conversions for an effect size in
non-multi-level studies, but we have not yet seen documentation on how to
do this sort of conversion for multi-level meta-analysis based on
regression outputs.

If the answer is a definitive ?No, this is not possible (at least at this
time)? ? this would also be a useful answer, as we have conducted
vote-counting tests in our paper, and can report on those results along
with the multi-level findings from the smaller number of effects that do
include the necessary information.

If the answer is ?Yes, this is possible but it?s complicated?? - any

advice

and/or references for addressing this issue would be greatly appreciated,
and we are happy to provide more information on the analysis as useful. We
are in the final stages of revising a paper that has received very

positive

feedback in various conferences (on the topic of political efficacy and
online political participation), and would be grateful for leads that

would

help us conduct as rigorous and comprehensive an analysis as possible.

Best wishes,
Jenny

Dr. Jennifer Oser
Department of Politics & Government
Ben-Gurion University of the Negev
https://www.jenniferoser.com/