[R-meta] Unrealistic confidence limits for heterogeneity?
Just a few comments 1 - when you say negative estimates of variance do you mean of tau^2? If so it is perfectly possible to get negative estimates of variance by choosing an appropriate estimator of tau^2. 2 - the Q-profile method of estimating conifdnce intervals about the estimate of tau^2 is explained in references in the documentation, probably here Jackson, D., Turner, R., Rhodes, K., & Viechtbauer, W. (2014). Methods for calculating confidence and credible intervals for the residual between-study variance in random effects meta-regression models. BMC Medical Research Methodology, 14, 103. ?https://doi.org/10.1186/1471-2288-14-103? 3 - I am not an expert on fitting mixed models in general in R but you need to specify which package you are using as there are a number of options, see the Task View https://cran.r-project.org/view=MixedModels for more details. Michael
On 30/03/2023 04:34, Will Hopkins via R-sig-meta-analysis wrote:
No replies from anyone as yet, so here is some additional info, another question, and further thoughts about negative variance arising from multiple within-study effects measured in the same subjects. The documentation for the way SAS generates confidence limits for variances and covariances in its mixed model is at this link: https://documentation.sas.com/doc/en/pgmsascdc/9.4_3.3/statug/statug_mixed_s yntax01.htm#statug.mixed.procstmt_cl . They are Wald limits for the variances when negative variance is allowed. It doesn't state there how the standard errors for the variances are estimated, but further down on that page under COVTEST (an option you have to select) it refers to "asymptotic standard errors". I asked in my previous message if the mixed models in R allow negative variance yet. I forgot to ask if they provide standard errors yet, too. Again, they weren't provided a few years ago. Someone who was helping me with R found some code that someone had written to generate standard errors, but the estimates were different from those of SAS, so I gave up on R for mixed modeling at that point. Further to multiple effects coming from the same subjects within studies resulting in negative variance for within-study heterogeneity... Multiple effects from the same subjects would be typically effects of multiple treatments in crossovers or effects of a treatment at different time points in a controlled trial, and they would be included in a meta-regression with appropriate fixed effects. The random error of measurement in such designs would not be correlated, but they would be correlated, if individual responses made substantial contributions to the measurement errors, and if the individual responses had some consistency across treatments or time point. In that case, the scatter of the means for each treatment or time point within each study, after fixed effects have been accounted for in the meta-regression, would be less than expected, given the standard error of the mean of each treatment or time point, so the variance would have to be negative (assuming there was no other source of within-study heterogeneity to offset the negative variance). In other words, negative variance would be evidence of consistent individual responses. In fact, if you change the sign and take the square root, I think you get an estimate of the consistent individual responses as a standard deviation. It's a lower limit for the individual responses, though, because there may be substantial positive unexplained within-study heterogeneity offsetting the negative variance. Also, I stated "I presume that combining the within- and between-study variances gives a realistic estimate of between-study heterogeneity, even when the within-study variance is negative." I'm probably wrong there: if you did separate metas for each treatment or time point, you would get correct estimates of the between-study heterogeneity, but I suspect that the average of those would be more than the sum of the positive between-study variance and negative within-study variance in the full meta model with two random effects, when there are consistent individual responses. I need to do some more simulations to check. I think the full model would give the most precise estimate of the mean effect, and correct precision for the modifiers in the model, because (hopefully) negative within-study variance correctly accounts for the repeated measurements on the same subjects. Will Hopkins https://sportsci.org https://sportsci.org/will
_______________________________________________ R-sig-meta-analysis mailing list @ R-sig-meta-analysis at r-project.org To manage your subscription to this mailing list, go to: https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis