No replies from anyone as yet, so here is some additional info, another question, and further thoughts about negative variance arising from multiple within-study effects measured in the same subjects. The documentation for the way SAS generates confidence limits for variances and covariances in its mixed model is at this link: https://documentation.sas.com/doc/en/pgmsascdc/9.4_3.3/statug/statug_mixed_s yntax01.htm#statug.mixed.procstmt_cl . They are Wald limits for the variances when negative variance is allowed. It doesn't state there how the standard errors for the variances are estimated, but further down on that page under COVTEST (an option you have to select) it refers to "asymptotic standard errors". I asked in my previous message if the mixed models in R allow negative variance yet. I forgot to ask if they provide standard errors yet, too. Again, they weren't provided a few years ago. Someone who was helping me with R found some code that someone had written to generate standard errors, but the estimates were different from those of SAS, so I gave up on R for mixed modeling at that point. Further to multiple effects coming from the same subjects within studies resulting in negative variance for within-study heterogeneity... Multiple effects from the same subjects would be typically effects of multiple treatments in crossovers or effects of a treatment at different time points in a controlled trial, and they would be included in a meta-regression with appropriate fixed effects. The random error of measurement in such designs would not be correlated, but they would be correlated, if individual responses made substantial contributions to the measurement errors, and if the individual responses had some consistency across treatments or time point. In that case, the scatter of the means for each treatment or time point within each study, after fixed effects have been accounted for in the meta-regression, would be less than expected, given the standard error of the mean of each treatment or time point, so the variance would have to be negative (assuming there was no other source of within-study heterogeneity to offset the negative variance). In other words, negative variance would be evidence of consistent individual responses. In fact, if you change the sign and take the square root, I think you get an estimate of the consistent individual responses as a standard deviation. It's a lower limit for the individual responses, though, because there may be substantial positive unexplained within-study heterogeneity offsetting the negative variance. Also, I stated "I presume that combining the within- and between-study variances gives a realistic estimate of between-study heterogeneity, even when the within-study variance is negative." I'm probably wrong there: if you did separate metas for each treatment or time point, you would get correct estimates of the between-study heterogeneity, but I suspect that the average of those would be more than the sum of the positive between-study variance and negative within-study variance in the full meta model with two random effects, when there are consistent individual responses. I need to do some more simulations to check. I think the full model would give the most precise estimate of the mean effect, and correct precision for the modifiers in the model, because (hopefully) negative within-study variance correctly accounts for the repeated measurements on the same subjects. Will Hopkins https://sportsci.org https://sportsci.org/will
[R-meta] Unrealistic confidence limits for heterogeneity?
3 messages · Will Hopkins, Michael Dewey, James Pustejovsky
Just a few comments 1 - when you say negative estimates of variance do you mean of tau^2? If so it is perfectly possible to get negative estimates of variance by choosing an appropriate estimator of tau^2. 2 - the Q-profile method of estimating conifdnce intervals about the estimate of tau^2 is explained in references in the documentation, probably here Jackson, D., Turner, R., Rhodes, K., & Viechtbauer, W. (2014). Methods for calculating confidence and credible intervals for the residual between-study variance in random effects meta-regression models. BMC Medical Research Methodology, 14, 103. ?https://doi.org/10.1186/1471-2288-14-103? 3 - I am not an expert on fitting mixed models in general in R but you need to specify which package you are using as there are a number of options, see the Task View https://cran.r-project.org/view=MixedModels for more details. Michael
On 30/03/2023 04:34, Will Hopkins via R-sig-meta-analysis wrote:
No replies from anyone as yet, so here is some additional info, another question, and further thoughts about negative variance arising from multiple within-study effects measured in the same subjects. The documentation for the way SAS generates confidence limits for variances and covariances in its mixed model is at this link: https://documentation.sas.com/doc/en/pgmsascdc/9.4_3.3/statug/statug_mixed_s yntax01.htm#statug.mixed.procstmt_cl . They are Wald limits for the variances when negative variance is allowed. It doesn't state there how the standard errors for the variances are estimated, but further down on that page under COVTEST (an option you have to select) it refers to "asymptotic standard errors". I asked in my previous message if the mixed models in R allow negative variance yet. I forgot to ask if they provide standard errors yet, too. Again, they weren't provided a few years ago. Someone who was helping me with R found some code that someone had written to generate standard errors, but the estimates were different from those of SAS, so I gave up on R for mixed modeling at that point. Further to multiple effects coming from the same subjects within studies resulting in negative variance for within-study heterogeneity... Multiple effects from the same subjects would be typically effects of multiple treatments in crossovers or effects of a treatment at different time points in a controlled trial, and they would be included in a meta-regression with appropriate fixed effects. The random error of measurement in such designs would not be correlated, but they would be correlated, if individual responses made substantial contributions to the measurement errors, and if the individual responses had some consistency across treatments or time point. In that case, the scatter of the means for each treatment or time point within each study, after fixed effects have been accounted for in the meta-regression, would be less than expected, given the standard error of the mean of each treatment or time point, so the variance would have to be negative (assuming there was no other source of within-study heterogeneity to offset the negative variance). In other words, negative variance would be evidence of consistent individual responses. In fact, if you change the sign and take the square root, I think you get an estimate of the consistent individual responses as a standard deviation. It's a lower limit for the individual responses, though, because there may be substantial positive unexplained within-study heterogeneity offsetting the negative variance. Also, I stated "I presume that combining the within- and between-study variances gives a realistic estimate of between-study heterogeneity, even when the within-study variance is negative." I'm probably wrong there: if you did separate metas for each treatment or time point, you would get correct estimates of the between-study heterogeneity, but I suspect that the average of those would be more than the sum of the positive between-study variance and negative within-study variance in the full meta model with two random effects, when there are consistent individual responses. I need to do some more simulations to check. I think the full model would give the most precise estimate of the mean effect, and correct precision for the modifiers in the model, because (hopefully) negative within-study variance correctly accounts for the repeated measurements on the same subjects. Will Hopkins https://sportsci.org https://sportsci.org/will
_______________________________________________ R-sig-meta-analysis mailing list @ R-sig-meta-analysis at r-project.org To manage your subscription to this mailing list, go to: https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
Some further comments in addition to Michael's response below. James 1. It is possible to allow for negative heterogeneity estimates using the metafor package. Here is an example of the syntax: library(metafor) # generate data with no heterogeneity set.seed(20230330) k <- 10 vi <- 4 / (rpois(k, 22) + 8) yi <- rnorm(k, mean = 0.2, sd = sqrt(vi)) dat <- data.frame(yi, vi) # regular random effects meta-analysis, REML estimator res1 <- rma(yi = yi, vi = vi, data=dat) res1 # allow negative heterogeneity, REML estimator res2 <- rma(yi = yi, vi = vi, data=dat, control=list(tau2.min=-min(vi))) res2 # allow negative heterogeneity, other heterogeneity estimators rma(yi = yi, vi = vi, data=dat, method = "ML", control=list(tau2.min=-min(vi))) rma(yi = yi, vi = vi, data=dat, method = "DL", control=list(tau2.min=-min(vi))) rma(yi = yi, vi = vi, data=dat, method = "HE", control=list(tau2.min=-min(vi))) 2. You can obtain the estimated standard error for tau-squared as follows: res1$se.tau2 res2$se.tau2 3. The metafor package implements several different confidence intervals for tau-squared. The GENQ method requires estimating the model with method GENQ. confint(res1) # confidence interval for tau-squared confint(res1, type = "PL") # profile likelihood method confint(res1, type = "QP") # Q-profile method rma(yi = yi, vi = vi, data=dat, weights = 1 / vi, method = "GENQ") |> confint(type = "GENQ") # Generalized Q-statistic method On Thu, Mar 30, 2023 at 5:32?AM Michael Dewey via R-sig-meta-analysis <
r-sig-meta-analysis at r-project.org> wrote:
Just a few comments 1 - when you say negative estimates of variance do you mean of tau^2? If so it is perfectly possible to get negative estimates of variance by choosing an appropriate estimator of tau^2. 2 - the Q-profile method of estimating conifdnce intervals about the estimate of tau^2 is explained in references in the documentation, probably here Jackson, D., Turner, R., Rhodes, K., & Viechtbauer, W. (2014). Methods for calculating confidence and credible intervals for the residual between-study variance in random effects meta-regression models. BMC Medical Research Methodology, 14, 103. ?https://doi.org/10.1186/1471-2288-14-103? 3 - I am not an expert on fitting mixed models in general in R but you need to specify which package you are using as there are a number of options, see the Task View https://cran.r-project.org/view=MixedModels for more details. Michael On 30/03/2023 04:34, Will Hopkins via R-sig-meta-analysis wrote:
No replies from anyone as yet, so here is some additional info, another question, and further thoughts about negative variance arising from
multiple
within-study effects measured in the same subjects. The documentation for the way SAS generates confidence limits for
variances
and covariances in its mixed model is at this link:
yntax01.htm#statug.mixed.procstmt_cl . They are Wald limits for the variances when negative variance is allowed. It doesn't state there how
the
standard errors for the variances are estimated, but further down on that page under COVTEST (an option you have to select) it refers to
"asymptotic
standard errors". I asked in my previous message if the mixed models in R allow negative variance yet. I forgot to ask if they provide standard errors yet, too. Again, they weren't provided a few years ago. Someone who was helping me with R found some code that someone had written to generate standard
errors,
but the estimates were different from those of SAS, so I gave up on R for mixed modeling at that point. Further to multiple effects coming from the same subjects within studies resulting in negative variance for within-study heterogeneity... Multiple effects from the same subjects would be typically effects of multiple treatments in crossovers or effects of a treatment at different time
points
in a controlled trial, and they would be included in a meta-regression
with
appropriate fixed effects. The random error of measurement in such
designs
would not be correlated, but they would be correlated, if individual responses made substantial contributions to the measurement errors, and
if
the individual responses had some consistency across treatments or time point. In that case, the scatter of the means for each treatment or time point within each study, after fixed effects have been accounted for in
the
meta-regression, would be less than expected, given the standard error of the mean of each treatment or time point, so the variance would have to
be
negative (assuming there was no other source of within-study
heterogeneity
to offset the negative variance). In other words, negative variance
would be
evidence of consistent individual responses. In fact, if you change the
sign
and take the square root, I think you get an estimate of the consistent individual responses as a standard deviation. It's a lower limit for the individual responses, though, because there may be substantial positive unexplained within-study heterogeneity offsetting the negative variance. Also, I stated "I presume that combining the within- and between-study variances gives a realistic estimate of between-study heterogeneity, even when the within-study variance is negative." I'm probably wrong there: if you did separate metas for each treatment or time point, you would get correct estimates of the between-study heterogeneity, but I suspect that
the
average of those would be more than the sum of the positive between-study variance and negative within-study variance in the full meta model with
two
random effects, when there are consistent individual responses. I need
to do
some more simulations to check. I think the full model would give the
most
precise estimate of the mean effect, and correct precision for the
modifiers
in the model, because (hopefully) negative within-study variance
correctly
accounts for the repeated measurements on the same subjects. Will Hopkins https://sportsci.org https://sportsci.org/will
_______________________________________________ R-sig-meta-analysis mailing list @ R-sig-meta-analysis at r-project.org To manage your subscription to this mailing list, go to: https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
-- Michael http://www.dewey.myzen.co.uk/home.html
_______________________________________________ R-sig-meta-analysis mailing list @ R-sig-meta-analysis at r-project.org To manage your subscription to this mailing list, go to: https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis