-----Original Message-----
From: Sotola, Lukas K [PSYCH] [mailto:lksotola at iastate.edu]
Sent: Tuesday, 13 June, 2023 17:45
To: Viechtbauer, Wolfgang (NP); R Special Interest Group for Meta-Analysis
Subject: RE: Non-independent effect sizes for moderator analysis in meta-analysis
on odds ratios
Hi Wolfgang,
Thank you for your quick response, which I found very helpful. With regards to
the final point, I would just like the meta-analytic effect size estimates for
both levels of my moderator expressed as odds ratios, rather than log odds
ratios. It is my understanding that when you run a meta-analysis using rma.mv(),
the estimates it yields are given as log odds ratios, and I would just like to
know if there is a way to back-transform those to odds ratios. The method you
pointed me to seems to yield predicted odds ratios for each of the individual
studies I analyze, but not to convert the overall meta-analytic effect size
estimates back to odds ratios.
Thank you,
Lukas
-----Original Message-----
From: Viechtbauer, Wolfgang (NP) <wolfgang.viechtbauer at maastrichtuniversity.nl>
Sent: Tuesday, June 13, 2023 1:58 AM
To: R Special Interest Group for Meta-Analysis <r-sig-meta-analysis at r-
project.org>
Cc: Sotola, Lukas K [PSYCH] <lksotola at iastate.edu>
Subject: RE: Non-independent effect sizes for moderator analysis in meta-analysis
on odds ratios
Dear Lukas,
You are asking about an issue that has been discussed quite extensively on this
mailing list, but let me repeat some of the relevant points:
If two odds ratios come from the same sample, then they are not independent.
Ignoring this dependency doesn't make your results "biased" (at least not in the
sense of how bias is typically defined in statistics); the real issue is that the
standard errors of the coefficients in the meta-regression model tend to be too
small, leading to inflated Type I error rates and confidence intervals that are
too narrow.
To deal with such dependency, one should ideally do several things:
1) Calculate the covariance between the dependent estimates. Just like we can
compute the sampling variance of each log odds ratio, we can also compute their
covariance. However, doing so if often tricky because the information needed to
compute the covariance is typically not reported. Alternatively, one can compute
an approximate covariance, making assumptions about the degree of dependency
between them (e.g., if the two log odds ratios are assessing a treatment effect
at two different timepoints, then they will tend to be more correlated if the two
timepoints are closer to each other; or if the two log odds ratios are assessing
a treatment effect for two different dichotomous response variables, then they
will tend to be more correlated if two variables are also strongly correlated).
One can use the vcalc() function to approximate the covariance making an
assumption about the degree of correlation. Typically, when 'guestimating' the
correlation, one also does a sensitivity analysis, assessing whether the
conclusions remain unchanged when different degrees of correlation are assumed.
2) In addition to 1), one should try to account for the dependency that may arise
in the underlying true effects (i.e., the true log odds ratios). This can be done
via a multilevel/multivariate model. This is what you have done with rma.mv().
3) Finally, one can consider using cluster-robust inference methods (also known
as robust variance estimation). However, with a small number of studies, this
might not work so well. Alternatively, one can consider using bootstrapping (see
https://doi.org/10.1002/jrsm.1554 and the wildmeta package).
See also:
https://wviechtb.github.io/metafor/reference/misc-recs.html#general-workflow-for-
meta-analyses-involving-complex-dependency-structures
As for your last question: Not sure what exactly you mean by "the results". Based
on the meta-regression model, you can compute predicted effects (log odds
ratios), which you can back-transform to odds ratios. This can be done with
predict(..., transf=exp).
Best,
Wolfgang
-----Original Message-----
From: R-sig-meta-analysis
[mailto:r-sig-meta-analysis-bounces at r-project.org] On Behalf Of Sotola,
Lukas K [PSYCH] via R-sig-meta-analysis
Sent: Monday, 12 June, 2023 17:01
To: r-sig-meta-analysis at r-project.org
Cc: Sotola, Lukas K [PSYCH]
Subject: [R-meta] Non-independent effect sizes for moderator analysis
in meta- analysis on odds ratios
Dear all,
I am trying to do a meta-analysis with odds ratios, but am running into
one issue. I have a few effect sizes that are not independent. That is,
there are a few samples in my data where we extracted two odds ratios
for the purposes of performing a moderator analysis. Thus, while we
have 6 samples (k = 6), we have 9 effect sizes, as three samples we
coded two odds ratios. Normally, when I run a meta-analysis with
correlation coefficients, there is some way to indicate a "Study ID"
variable of some sort, which will allow the analysis to distinguish
effect sizes taken from the same sample for a moderator analysis. That way,
multiple effect sizes from the same sample do not get counted multiple times.
However, I cannot seem to find an analogous feature using metafor, and
even when I run the meta-analysis with a moderator analysis, in the
overall analysis (e.g., for heterogeneity and such), the output
indicates a "k" that is equal to the number of effect sizes and not equal to the
real number of samples (i.e., k = 9).
This makes me worry that the results I'm getting are biased.
After using the "escalc" function to create a new dataset with odds
ratios as shown on the metafor website, I have attempted the analysis
itself in two ways so far. The name of my moderator variable is
"HighACEDef" and it has two levels. For my first attempt at the
meta-analysis, I use the following R code and get the output that follows it:
Analysis1 <- rma(yi, vi, mods = ~factor(HighACEDef), data=dat1)
Analysis1
Mixed-Effects Model (k = 9; tau^2 estimator: REML)
tau^2 (estimated amount of residual heterogeneity): 0.0252 (SE = 0.0203)
tau (square root of estimated tau^2 value): 0.1587
I^2 (residual heterogeneity / unaccounted variability): 82.50%
H^2 (unaccounted variability / sampling variability): 5.71
R^2 (amount of heterogeneity accounted for): 0.00%
Test for Residual Heterogeneity:
QE(df = 7) = 33.6446, p-val < .0001
Test of Moderators (coefficient 2):
QM(df = 1) = 0.7218, p-val = 0.3955
Model Results:
estimate se zval pval ci.lb ci.ub
intrcpt 0.8144 0.0853 9.5439 <.0001 0.6472 0.9817 ***
factor(HighACEDef)>=4 0.1151 0.1355 0.8496 0.3955 -0.1505 0.3807
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
The second way I have tried it is with this code and I get the output
that follows it. "ES_ID" is a unique identification number for each
separate effect size, whereas "ID" is a unique identification number
for each sample we coded from. Thus, "ES_ID" has the values of 1-9 while "ID"
Analysis2 <- rma.mv(yi, vi, random = ~ ES_ID | ID, mods =
~factor(HighACEDef),
data=dat1)
Analysis2
Multivariate Meta-Analysis Model (k = 9; method: REML)
Variance Components:
outer factor: ID (nlvls = 6)
inner factor: ES_ID (nlvls = 9)
estim sqrt fixed
tau^2 0.0302 0.1739 no
rho 0.6011 no
Test for Residual Heterogeneity:
QE(df = 7) = 33.6446, p-val < .0001
Test of Moderators (coefficient 2):
QM(df = 1) = 0.8522, p-val = 0.3559
Model Results:
estimate se zval pval ci.lb ci.ub
intrcpt 0.8213 0.0914 8.9849 <.0001 0.6421 1.0004 ***
factor(HighACEDef)>=4 0.0967 0.1047 0.9232 0.3559 -0.1086 0.3020
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
I would much appreciate any help with doing this correctly, or just a
confirmation that either of the ways I have already done it is correct.
I also had one last question. Is there any way the results can be
expressed as odds ratios rather than as log odds ratios?
Thank you,
Lukas Sotola
Lukas K. Sotola, M.S.
PhD Candidate: Iowa State University
Department of Psychology
He/him