[R-meta] Unrealistic confidence limits for heterogeneity?

Sun, Apr 2, 2023 7:02 AM

Interesting point.

Just to note: Right now, a warning is already issued when tau2.min is set to a value lower than -min(vi) (as it then automatically sets tau2.min to -min(vi)). This is also documented (https://wviechtb.github.io/metafor/reference/rma.uni.html#note-1):

rma(yi, vi, data=dat, control=list(tau2.min=-1))
[...]
Warning message:
Value of 'tau2.min' constrained to -min(vi) = -0.1212.

Also, the function prevents shenanigans like trying to force tau^2 to be smaller than -min(vi):

rma(yi, vi, data=dat, tau2=-1)

Error in rma(yi, vi, data = dat, tau2 = -1) :
  Some marginal variances are negative.

But if the *estimate* is equal to tau2.min, then there is no further note about this (except that the user will see that in the output). In a way, this is no different than when tau^2 is estimated to be 0 (under the default tau2.min=0 setting).

Technically, it is not even possible to fit the model with tau2 = -min(vi) with ML/REML estimation, since this will lead to division by zero:

rma(yi, vi, data=dat, tau2=-min(vi))

Error in rma(yi, vi, data = dat, tau2 = -min(vi)) :
  Division by zero when computing the inverse variance weights.

The step halving algorithm that is used as part of ML/REML estimation in rma.uni() should essentially prevent that the estimate can actually hit -min(vi).

One can of course construct a dataset where the likelihood profile peaks essentially at -min(vi) (or rather, just a smidge above it), which might suggest that a value of tau^2 that is even smaller than -min(vi) could lead to an even larger likelihood, but then one would have to allow negative weights.

Best,
Wolfgang

-----Original Message-----
From: Michael Dewey [mailto:lists at dewey.myzen.co.uk]
Sent: Sunday, 02 April, 2023 12:10
To: R Special Interest Group for Meta-Analysis
Cc: Viechtbauer, Wolfgang (NP)
Subject: Re: [R-meta] Unrealistic confidence limits for heterogeneity?

This raises the question of what to do when an estimate is at the
boundary of the parameter space (or beyond it). Is it worth warning the
user that this may have happened (ie the lower bound is at -min(vi)? It
suggests to me that the user has here an attempt to fit a model to a
dataset which is inadequate for that model.

Michael

On 01/04/2023 13:17, Viechtbauer, Wolfgang (NP) via R-sig-meta-analysis
wrote:

Hi James,

Your example made me realize some tricky issues when tau2.min is set to -

min(vi) (which is the smallest value of tau2.min allowed, since otherwise vi +
tau^2 could become negative) and one then wants to compute CIs for tau^2 (in your
example, confint() fails with res2). In the devel version, I now push tau2.min
just slightly above -min(vi) in confint() that hopefully deals with this issue (a
bit of a hack but seems to work, at least in your example).

res2 <- rma(yi, vi, data=dat, control=list(tau2.min=-min(vi)))
res2
confint(res2)
confint(res2, type="PL")
res2 <- rma(yi, vi, data=dat, method="GENQ", weights=1/vi,

control=list(tau2.min=-min(vi)))

confint(res2)

Best,
Wolfgang

-----Original Message-----
From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces at r-project.org]

On

Behalf Of James Pustejovsky via R-sig-meta-analysis
Sent: Thursday, 30 March, 2023 19:00
To: R Special Interest Group for Meta-Analysis
Cc: James Pustejovsky
Subject: Re: [R-meta] Unrealistic confidence limits for heterogeneity?

Some further comments in addition to Michael's response below.

James

1. It is possible to allow for negative heterogeneity estimates using the
metafor package. Here is an example of the syntax:

library(metafor)

# generate data with no heterogeneity
set.seed(20230330)
k <- 10
vi <- 4 / (rpois(k, 22) + 8)
yi <- rnorm(k, mean = 0.2, sd = sqrt(vi))
dat <- data.frame(yi, vi)

# regular random effects meta-analysis, REML estimator
res1 <- rma(yi = yi, vi = vi, data=dat)
res1

# allow negative heterogeneity, REML estimator
res2 <- rma(yi = yi, vi = vi, data=dat, control=list(tau2.min=-min(vi)))
res2

# allow negative heterogeneity, other heterogeneity estimators
rma(yi = yi, vi = vi, data=dat, method = "ML",
control=list(tau2.min=-min(vi)))
rma(yi = yi, vi = vi, data=dat, method = "DL",
control=list(tau2.min=-min(vi)))
rma(yi = yi, vi = vi, data=dat, method = "HE",
control=list(tau2.min=-min(vi)))

2. You can obtain the estimated standard error for tau-squared as follows:
res1$se.tau2
res2$se.tau2

3. The metafor package implements several different confidence intervals
for tau-squared. The GENQ method requires estimating the model with method
GENQ.
confint(res1) # confidence interval for tau-squared
confint(res1, type = "PL") # profile likelihood method
confint(res1, type = "QP") # Q-profile method
rma(yi = yi, vi = vi, data=dat, weights = 1 / vi, method = "GENQ")  |>
  confint(type = "GENQ") # Generalized Q-statistic method

[R-meta] Unrealistic confidence limits for heterogeneity?

Thread (5 messages)