[R-meta] Credibility intervals with RVE

Hi all,

Credibility intervals can be useful because ? in contrast to confidence
intervals ? they consider the heterogeneity of synthesized effects and make
predictions about the range of values within which the true effect of
future studies will likely fall (e.g., IntHout et al., 2016).

It is straightforward enough to calculate credibility intervals for rma and
rma.mv objects in metafor using the `predict` function. However, (a) how
would one compute credibility intervals for multivariate multilevel
meta-analytic models with cluster-robust variance estimation, and (b) is
doing so even meaningful, given that the covariance matrix has only been
aproximated? I included more details about both questions below.

Thanks a lot!
Fabian

-----

Regarding question (a), I would compute the credibility interval for a
simplified example as follows:

# fit model
vcv <- clubSandwich::impute_covariance_matrix(vi = data$vi, cluster =
data$cluster_id, r = 0.7)
m <- metafor::rma.mv(yi, vcv, random = ~ 1 | cluster_id/es_id, data =
data)
robu <- clubSandwich::coef_test(m, vcov = "CR2")

# compute 95% credibility interval
robu$beta - 1.96*sqrt(sum(m$sigma2) + robu$SE^2)
robu$beta + 1.96*sqrt(sum(m$sigma2) + robu$SE^2)

Regarding question (b), since V in this model is just a rough approximation
of the empirical covariance structure, the variance components of the model
are also just an approximation as well. Even when using cluster-robust
standard errors for the computation of the credibility intervals, we still
use the model's approximated variance components, and thus the credibility
interval provides just a rough guess of the range within which future
studies will fall.

        [[alternative HTML version deleted]]

Fabian,

a) Your approach is exactly what I would do (and have done in the past).
b) You are exactly right about the limitations of the approach. With RVE,
it's recommended to conduct sensitivity analyses. You could do the same
thing here, and report how the prediction interval changes depending on the
assumed value of r.

James

On Tue, Apr 9, 2019 at 1:44 PM Fabian Schellhaas <
fabian.schellhaas at yale.edu> wrote:

Hi all,

Credibility intervals can be useful because ? in contrast to confidence
intervals ? they consider the heterogeneity of synthesized effects and
make
predictions about the range of values within which the true effect of
future studies will likely fall (e.g., IntHout et al., 2016).

It is straightforward enough to calculate credibility intervals for rma
and
rma.mv objects in metafor using the `predict` function. However, (a) how
would one compute credibility intervals for multivariate multilevel
meta-analytic models with cluster-robust variance estimation, and (b) is
doing so even meaningful, given that the covariance matrix has only been
aproximated? I included more details about both questions below.

Thanks a lot!
Fabian

-----

Regarding question (a), I would compute the credibility interval for a
simplified example as follows:

# fit model
vcv <- clubSandwich::impute_covariance_matrix(vi = data$vi, cluster =
data$cluster_id, r = 0.7)
m <- metafor::rma.mv(yi, vcv, random = ~ 1 | cluster_id/es_id, data =
data)
robu <- clubSandwich::coef_test(m, vcov = "CR2")

# compute 95% credibility interval
robu$beta - 1.96*sqrt(sum(m$sigma2) + robu$SE^2)
robu$beta + 1.96*sqrt(sum(m$sigma2) + robu$SE^2)

Regarding question (b), since V in this model is just a rough
approximation
of the empirical covariance structure, the variance components of the
model
are also just an approximation as well. Even when using cluster-robust
standard errors for the computation of the credibility intervals, we still
use the model's approximated variance components, and thus the credibility
interval provides just a rough guess of the range within which future
studies will fall.

        [[alternative HTML version deleted]]