[R-meta] Notable difference between treditional and bootstrap 95% CI for sigma2: which one is preffered?

On question (3) regarding I-squared with RVE, I agree with Wolfgang's 
initial assessment that RVE and I-squared pertain to different aspects 
of the model. RVE is about assessing uncertainty in the average effect 
size or (more generally) meta-regression coefficients of the model (the 
"fixed" part of the model). It is robust in the sense that it does not 
require correct specification of the random effects structure or 
sampling variances/covariances (the "random" part of the model).

I-squared is a description of the proportion of variation in effect 
size estimates that can be attributed to true heterogeneity in effect 
sizes. Its definition is contingent on the assumptions about the random 
effects in the model, which are not "robust" in the RVE sense.

On Tue, Mar 29, 2022 at 6:45 AM Viechtbauer, Wolfgang (SP) 
<wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:

Dear Ali,

1) I would say not much. confint() gives you a profile likelihood CI, 
bootstrapping a different type of CI. I wouldn't expect them to be 
similar in the first place - maybe asymptotically, but not even sure 
about that.

I examined profile likelihood versus bootstrap (versus a bunch of 
other) CIs for the simpler standard RE model in this paper:

Viechtbauer, W. (2007). Confidence intervals for the amount of 
heterogeneity in meta-analysis. Statistics in Medicine, 26(1), 37-52. 
https://doi.org/10.1002/sim.2514

At least in this case, the bootstrap CIs didn't fare so well. The 
profile likelihood CIs did better although they are based on 
large-sample theory, so if k is small, then not so great either (and 
with log odds ratios - as examined in the paper above - things go 
really bad when the within-study sample sizes are small, since the 
estimated sampling variances can then be really off).

2) For the moment, I would go with the profile ll CIs.

3) Hmmm, that's a tricky one. In principle, the I^2 calculation and 
RVE are about different things. I^2 is asking how much of the total 
variance is due to heterogeneity (or particular variance components in 
the model), while RVE is about making inferences about the model 
coefficients. But RVE is also in some sense about the variance -- it 
uses the product of the residuals to get a (very rough!) approximation 
to the marginal var-cov matrix of the effect size estimates and then 
squishes this together into the var-cov matrix of the model 
coefficients (which then ends up being a really good approximation to 
the var-cov matrix of the model coefficients). Maybe one could compute 
a sort of robust version of the P matrix that is used in the 
calculation of I^2 - which might again be a very rough approximation, 
but since I^2 in essence takes the average of the trace of P, this 
'cluster-robust version of P' might again be acceptable to use in the 
calculation of I^2. But all of this is just mere brainstorming. At the 
moment, I would just report the I^2 from the model before applying 
RVE.

Best,
Wolfgang

-----Original Message-----
From: R-sig-meta-analysis 
[mailto:r-sig-meta-analysis-bounces at r-project.org] On
Behalf Of towhidi
Sent: Monday, 28 March, 2022 12:54
To: r sig meta-analysis list
Subject: [R-meta] Notable difference between treditional and 
bootstrap 95% CI for
sigma2: which one is preffered?

Dear all,

I am working on a dataset with a multilevel structure: 185 SMDs, 
nested
in 108 outcomes, nested in 41 comparisons (to address multiarmed 
trials)
nested in 34 studies (random = ~1 |
stud_id/cont_id/outcome_id/occasion).

For some of the sigma^2 values, the CI from confint() is largely
different from the bootstrap CI, e.g., for a sigma^2 = .04, the upper
limit from confint() is .38, while the boot CI upper limit is .21.

(1) What does this difference imply?

(2) When such differences exist between traditional and boot CIs, 
Which
one is more reliable?

For calculating boot CI I used the following:

sim <- simulate(res, nsim=300)
sav <- lapply(sim, function(x) {
tmp <- try(rma.mv(x, vi, data = dat, random = res$random), 
silent=TRUE)
if (inherits(tmp, "try-error")) {
next
} else {
tmp
}})

sigma2.l4 <- sapply(sav, function(x) x$sigma2[2])

quantile(sigma2.l4, c(0.025, .975))

Of note, I have checked the profile plot and there seemed to be no
convergence problem.

I also have another related question:
(3) Is the general formula for I^2 for multilevel models
(https://www.metafor-project.org/doku.php/tips:i2_multilevel_multivariate)
can be applied to RVE without any modifications?

Thank you.

--
Ali Zia-Tohidi MSc
Clinical Psychology
University of Tehran