[R-meta] Unrealistic confidence limits for heterogeneity?

James and Michael, thanks for your replies.  Yes, I meant negative values for the variance representing heterogeneity. Good to see that you can allow for such negative variance in metafor (and thanks for all that code, James). 

I'd still like to hear if negative variance is possible with the mixed model(s) in R, and whether it gives standard errors for the variances. I was trying out mixed modeling with lme4, as documented at https://www.sportsci.org/2016/inbrief.htm#SAS . There's a link there to some R programs and instructions provided by the person who was showing me how to use R, Alice Sweeting. It's in the paragraph below all the updates.

I still need feedback from anyone with an opinion on the argument for using negative variance, and feedback on the issue of the claim that there is downward bias in the estimate of heterogeneity. Someone called Donald Williams made that claim at https://psyarxiv.com/7tbrm/ : "with increasing values of tau^2, however, the DL estimator can show substantial downward bias... A number of studies have shown that the REML method shows less downward bias in the estimate of between study variance than the DL estimator (Chung, Rabe-Hesketh, & Choi, 2013; Sidik & Jonkman, 2007)". That's as may be, but the estimator in SAS is REML, and in my hands it gives unbiased estimates of the heterogeneity variance (tau^2), regardless of the number of studies, standard error of the study-estimates, and the true heterogeneity. Williams shows graphs only of tau, not tau^2. It  looks to me like the downward bias is simply small-DF bias, and you can correct it with the 1+1/(4DF) factor. Using Bayesian priors is not the way to correct it. If you want to make inferences about the magnitude of the heterogeneity relative to smallest and other important threshold values, square the threshold values and assume the variance has a t distribution, thereby avoiding the problem of small-DF bias in tau and partly offsetting the problem of the confidence interval being too narrow. (The threshold values for tau are one half the threshold values for the mean, by the way.)

I've started doing simulations of metas with multiple within-study effects from the same subjects in whom there are consistent individual responses. Yes, it does give negative variance for the within-study heterogeneity, but some of the individual-response variance adds to the between-study variance, which makes sense when you think about it. Also, the negative variance and the extra between-study variance are of the order of the individual response variance divided by the square root of the study sample size, so it's going to be a mess when sample sizes vary. Unfortunately the confidence interval for the mean effect is too wide, so for now, I can't do metas of multiple effects, if the effects include substantial consistent individual responses. I'm hoping it will be possible to somehow include and thereby adjust for the estimate of individual-response variance in each study (for a controlled trial, it's the difference in the variances of the change scores in the experimental and control groups), but I can't yet see how to do it. Maybe with some kind of spatial covariance structure.

Will
https://sportsci.org
https://sportsci.org/will

-----Original Message-----
From: R-sig-meta-analysis <r-sig-meta-analysis-bounces at r-project.org> On Behalf Of James Pustejovsky via R-sig-meta-analysis
Sent: Friday, March 31, 2023 6:00 AM
To: R Special Interest Group for Meta-Analysis <r-sig-meta-analysis at r-project.org>
Cc: James Pustejovsky <jepusto at gmail.com>
Subject: Re: [R-meta] Unrealistic confidence limits for heterogeneity?

Some further comments in addition to Michael's response below.

James

1. It is possible to allow for negative heterogeneity estimates using the metafor package. Here is an example of the syntax:

library(metafor)

# generate data with no heterogeneity
set.seed(20230330)
k <- 10
vi <- 4 / (rpois(k, 22) + 8)
yi <- rnorm(k, mean = 0.2, sd = sqrt(vi)) dat <- data.frame(yi, vi)

# regular random effects meta-analysis, REML estimator
res1 <- rma(yi = yi, vi = vi, data=dat)
res1

# allow negative heterogeneity, REML estimator
res2 <- rma(yi = yi, vi = vi, data=dat, control=list(tau2.min=-min(vi)))
res2

# allow negative heterogeneity, other heterogeneity estimators rma(yi = yi, vi = vi, data=dat, method = "ML",
control=list(tau2.min=-min(vi)))
rma(yi = yi, vi = vi, data=dat, method = "DL",
control=list(tau2.min=-min(vi)))
rma(yi = yi, vi = vi, data=dat, method = "HE",
control=list(tau2.min=-min(vi)))

2. You can obtain the estimated standard error for tau-squared as follows:
res1$se.tau2
res2$se.tau2

3. The metafor package implements several different confidence intervals for tau-squared. The GENQ method requires estimating the model with method GENQ.
confint(res1) # confidence interval for tau-squared confint(res1, type = "PL") # profile likelihood method confint(res1, type = "QP") # Q-profile method rma(yi = yi, vi = vi, data=dat, weights = 1 / vi, method = "GENQ")  |>
  confint(type = "GENQ") # Generalized Q-statistic method