[R-meta] Notable difference between treditional and bootstrap 95% CI for sigma2: which one is preffered?

Dear Ali,

1) I would say not much. confint() gives you a profile likelihood CI, bootstrapping a different type of CI. I wouldn't expect them to be similar in the first place - maybe asymptotically, but not even sure about that.

I examined profile likelihood versus bootstrap (versus a bunch of other) CIs for the simpler standard RE model in this paper:

Viechtbauer, W. (2007). Confidence intervals for the amount of heterogeneity in meta-analysis. Statistics in Medicine, 26(1), 37-52. https://doi.org/10.1002/sim.2514 

At least in this case, the bootstrap CIs didn't fare so well. The profile likelihood CIs did better although they are based on large-sample theory, so if k is small, then not so great either (and with log odds ratios - as examined in the paper above - things go really bad when the within-study sample sizes are small, since the estimated sampling variances can then be really off).

2) For the moment, I would go with the profile ll CIs.

3) Hmmm, that's a tricky one. In principle, the I^2 calculation and RVE are about different things. I^2 is asking how much of the total variance is due to heterogeneity (or particular variance components in the model), while RVE is about making inferences about the model coefficients. But RVE is also in some sense about the variance -- it uses the product of the residuals to get a (very rough!) approximation to the marginal var-cov matrix of the effect size estimates and then squishes this together into the var-cov matrix of the model coefficients (which then ends up being a really good approximation to the var-cov matrix of the model coefficients). Maybe one could compute a sort of robust version of the P matrix that is used in the calculation of I^2 - which might again be a very rough approximation, but since I^2 in essence takes the average of the trace of P, this 'cluster-robust version of P' might again be acceptable to use in the calculation of I^2. But all of this is just mere brainstorming. At the moment, I would just report the I^2 from the model before applying RVE.

Best,
Wolfgang

-----Original Message-----
From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces at r-project.org] On
Behalf Of towhidi
Sent: Monday, 28 March, 2022 12:54
To: r sig meta-analysis list
Subject: [R-meta] Notable difference between treditional and bootstrap 95% CI for
sigma2: which one is preffered?

Dear all,

I am working on a dataset with a multilevel structure: 185 SMDs, nested
in 108 outcomes, nested in 41 comparisons (to address multiarmed trials)
nested in 34 studies (random = ~1 |
stud_id/cont_id/outcome_id/occasion).

For some of the sigma^2 values, the CI from confint() is largely
different from the bootstrap CI, e.g., for a sigma^2 = .04, the upper
limit from confint() is .38, while the boot CI upper limit is .21.

(1) What does this difference imply?

(2) When such differences exist between traditional and boot CIs, Which
one is more reliable?

For calculating boot CI I used the following:

sim <- simulate(res, nsim=300)
sav <- lapply(sim, function(x) {
tmp <- try(rma.mv(x, vi, data = dat, random = res$random), silent=TRUE)
if (inherits(tmp, "try-error")) {
next
} else {
tmp
}})

sigma2.l4 <- sapply(sav, function(x) x$sigma2[2])

quantile(sigma2.l4, c(0.025, .975))

Of note, I have checked the profile plot and there seemed to be no
convergence problem.

I also have another related question:
(3) Is the general formula for I^2 for multilevel models
(https://www.metafor-project.org/doku.php/tips:i2_multilevel_multivariate)
can be applied to RVE without any modifications?

Thank you.

--
Ali Zia-Tohidi MSc
Clinical Psychology
University of Tehran