[R-meta] Comparing heterogeneity explained from different meta-regression models with the same moderator

Dear Zhenyu,

Making inferences about the (pseudo) R^2 statistic is a bit tricky, since it is computed by essentially comparing the tau^2 estimate from the random-effects model and the mixed-effects meta-regression model, so two different models. Hence, computing some kind of standard error for R^2 would be quite tricky. To get a CI for R^2, one could use bootstrapping. For example:

library(metafor)
library(boot)

dat <- dat.bangertdrowns2004
dat <- dat[!is.na(dat$minutes),]

rma(yi, vi, mods = ~ minutes, data=dat)  

boot.func <- function(dat, indices) {
   res <- try(suppressWarnings(rma(yi, vi, mods = ~ minutes, data=dat[indices,])), silent=TRUE)
   ifelse(inherits(res, "try-error"), NA, res$R2)
}

set.seed(1234)
res.boot <- boot(dat, boot.func, R=10000)
res.boot

hist(res.boot$t, breaks=50)

boot.ci(res.boot)

You will see that the bootstrap distribution of R^2 is quite weird, so I would recommend the percentile CI (which also avoids the possibility of getting bounds below 0% or above 100%). And the bootstrap CIs are super wide. That's to be expected when k is 'only' 24. Precise estimates of R^2 require quite large datasets.

The same applies if one would want to compare R^2 from two different models. The only way I could think of doing this is to use bootstrapping. How to set this up requires some thought. Are the two outcomes coming from the same set of studies? In that case, one might want to sample studies, not estimates. Then calculate R^2 for the two different models, take the difference, and repeat. If the data are in wide format (so the two outcomes are different columns and so you have one row per study), then this should work:

boot.func <- function(dat, indices) {
   res1 <- try(suppressWarnings(rma(y1i, v1i, mods = ~ x, data=dat[indices,])), silent=TRUE)
   res2 <- try(suppressWarnings(rma(y2i, v2i, mods = ~ x, data=dat[indices,])), silent=TRUE)
   ifelse(inherits(res1, "try-error") || inherits(res2, "try-error"), NA, res1$R2-res2$R2)
}

Best,
Wolfgang

-----Original Message-----
From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces at r-project.org] On
Behalf Of Yuan, Zhenyu
Sent: Tuesday, 20 September, 2022 2:32
To: r-sig-meta-analysis at r-project.org
Subject: [R-meta] Comparing heterogeneity explained from different meta-
regression models with the same moderator

Greetings,

I ran into a question regarding the heterogeneity explained by a meta-regression
model and I would appreciate any insights or guidance you all can provide.

Specifically, I have two meta-regression models (analyzed via the "metafor"
package) whereby the outcome variables are different but the moderator is the
same. I reported the heterogeneity explained in these two models in my research.
I was asked whether there would be any way to formally test the difference in the
amount of heterogeneity explained between these two models. My understanding is
that this would be difficult given that the outcome variables are different.
However, I wanted to reach out to the listserv and see if my understanding is
correct.

I'd greatly appreciate any guidance in this regard. Thank you for your time.

Best,

Zhenyu Yuan
Assistant Professor
Department of Managerial Studies
University of Illinois Chicago