[R-meta] Non-independent effect sizes for moderator analysis in meta-analysis on odds ratios
Hi Wolfgang, Thank you for your quick response, which I found very helpful. With regards to the final point, I would just like the meta-analytic effect size estimates for both levels of my moderator expressed as odds ratios, rather than log odds ratios. It is my understanding that when you run a meta-analysis using rma.mv(), the estimates it yields are given as log odds ratios, and I would just like to know if there is a way to back-transform those to odds ratios. The method you pointed me to seems to yield predicted odds ratios for each of the individual studies I analyze, but not to convert the overall meta-analytic effect size estimates back to odds ratios. Thank you, Lukas -----Original Message----- From: Viechtbauer, Wolfgang (NP) <wolfgang.viechtbauer at maastrichtuniversity.nl> Sent: Tuesday, June 13, 2023 1:58 AM To: R Special Interest Group for Meta-Analysis <r-sig-meta-analysis at r-project.org> Cc: Sotola, Lukas K [PSYCH] <lksotola at iastate.edu> Subject: RE: Non-independent effect sizes for moderator analysis in meta-analysis on odds ratios Dear Lukas, You are asking about an issue that has been discussed quite extensively on this mailing list, but let me repeat some of the relevant points: If two odds ratios come from the same sample, then they are not independent. Ignoring this dependency doesn't make your results "biased" (at least not in the sense of how bias is typically defined in statistics); the real issue is that the standard errors of the coefficients in the meta-regression model tend to be too small, leading to inflated Type I error rates and confidence intervals that are too narrow. To deal with such dependency, one should ideally do several things: 1) Calculate the covariance between the dependent estimates. Just like we can compute the sampling variance of each log odds ratio, we can also compute their covariance. However, doing so if often tricky because the information needed to compute the covariance is typically not reported. Alternatively, one can compute an approximate covariance, making assumptions about the degree of dependency between them (e.g., if the two log odds ratios are assessing a treatment effect at two different timepoints, then they will tend to be more correlated if the two timepoints are closer to each other; or if the two log odds ratios are assessing a treatment effect for two different dichotomous response variables, then they will tend to be more correlated if two variables are also strongly correlated). One can use the vcalc() function to approximate the covariance making an assumption about the degree of correlation. Typically, when 'guestimating' the correlation, one also does a sensitivity analysis, assessing whether the conclusions remain unchanged when different degrees of correlation are assumed. 2) In addition to 1), one should try to account for the dependency that may arise in the underlying true effects (i.e., the true log odds ratios). This can be done via a multilevel/multivariate model. This is what you have done with rma.mv(). 3) Finally, one can consider using cluster-robust inference methods (also known as robust variance estimation). However, with a small number of studies, this might not work so well. Alternatively, one can consider using bootstrapping (see https://doi.org/10.1002/jrsm.1554 and the wildmeta package). See also: https://wviechtb.github.io/metafor/reference/misc-recs.html#general-workflow-for-meta-analyses-involving-complex-dependency-structures As for your last question: Not sure what exactly you mean by "the results". Based on the meta-regression model, you can compute predicted effects (log odds ratios), which you can back-transform to odds ratios. This can be done with predict(..., transf=exp). Best, Wolfgang
-----Original Message-----
From: R-sig-meta-analysis
[mailto:r-sig-meta-analysis-bounces at r-project.org] On Behalf Of Sotola,
Lukas K [PSYCH] via R-sig-meta-analysis
Sent: Monday, 12 June, 2023 17:01
To: r-sig-meta-analysis at r-project.org
Cc: Sotola, Lukas K [PSYCH]
Subject: [R-meta] Non-independent effect sizes for moderator analysis
in meta- analysis on odds ratios
Dear all,
I am trying to do a meta-analysis with odds ratios, but am running into
one issue. I have a few effect sizes that are not independent. That is,
there are a few samples in my data where we extracted two odds ratios
for the purposes of performing a moderator analysis. Thus, while we
have 6 samples (k = 6), we have 9 effect sizes, as three samples we
coded two odds ratios. Normally, when I run a meta-analysis with
correlation coefficients, there is some way to indicate a "Study ID"
variable of some sort, which will allow the analysis to distinguish
effect sizes taken from the same sample for a moderator analysis. That way, multiple effect sizes from the same sample do not get counted multiple times.
However, I cannot seem to find an analogous feature using metafor, and
even when I run the meta-analysis with a moderator analysis, in the
overall analysis (e.g., for heterogeneity and such), the output
indicates a "k" that is equal to the number of effect sizes and not equal to the real number of samples (i.e., k = 9).
This makes me worry that the results I'm getting are biased.
After using the "escalc" function to create a new dataset with odds
ratios as shown on the metafor website, I have attempted the analysis
itself in two ways so far. The name of my moderator variable is
"HighACEDef" and it has two levels. For my first attempt at the
meta-analysis, I use the following R code and get the output that follows it:
Analysis1 <- rma(yi, vi, mods = ~factor(HighACEDef), data=dat1)
Analysis1
Mixed-Effects Model (k = 9; tau^2 estimator: REML)
tau^2 (estimated amount of residual heterogeneity): 0.0252 (SE = 0.0203)
tau (square root of estimated tau^2 value): 0.1587
I^2 (residual heterogeneity / unaccounted variability): 82.50%
H^2 (unaccounted variability / sampling variability): 5.71
R^2 (amount of heterogeneity accounted for): 0.00%
Test for Residual Heterogeneity:
QE(df = 7) = 33.6446, p-val < .0001
Test of Moderators (coefficient 2):
QM(df = 1) = 0.7218, p-val = 0.3955
Model Results:
estimate se zval pval ci.lb ci.ub
intrcpt 0.8144 0.0853 9.5439 <.0001 0.6472 0.9817 ***
factor(HighACEDef)>=4 0.1151 0.1355 0.8496 0.3955 -0.1505 0.3807
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
The second way I have tried it is with this code and I get the output
that follows it. "ES_ID" is a unique identification number for each
separate effect size, whereas "ID" is a unique identification number
for each sample we coded from. Thus, "ES_ID" has the values of 1-9 while "ID" has the values of 1-6.
Analysis2 <- rma.mv(yi, vi, random = ~ ES_ID | ID, mods =
~factor(HighACEDef),
data=dat1)
Analysis2
Multivariate Meta-Analysis Model (k = 9; method: REML)
Variance Components:
outer factor: ID (nlvls = 6)
inner factor: ES_ID (nlvls = 9)
estim sqrt fixed
tau^2 0.0302 0.1739 no
rho 0.6011 no
Test for Residual Heterogeneity:
QE(df = 7) = 33.6446, p-val < .0001
Test of Moderators (coefficient 2):
QM(df = 1) = 0.8522, p-val = 0.3559
Model Results:
estimate se zval pval ci.lb ci.ub
intrcpt 0.8213 0.0914 8.9849 <.0001 0.6421 1.0004 ***
factor(HighACEDef)>=4 0.0967 0.1047 0.9232 0.3559 -0.1086 0.3020
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
I would much appreciate any help with doing this correctly, or just a
confirmation that either of the ways I have already done it is correct.
I also had one last question. Is there any way the results can be
expressed as odds ratios rather than as log odds ratios?
Thank you,
Lukas Sotola
Lukas K. Sotola, M.S.
PhD Candidate: Iowa State University
Department of Psychology
He/him