Metafor multilevel metaregression: total variance increases when moderator added?

Tue, Feb 28, 2017 4:54 AM

Very difficult to diagnose what is going on without actually seeing the data. But as I said on CV: Depending on the data, the variance components may not be estimated precisely, so negative values for those kinds of pseudo-R^2 statistics are quite possible. In fact, if a particular moderator is actually unrelated to the outcomes, then in roughly 50% of the cases, the pseudo-R^2 statistic will be negative.

See also:

Lopez-Lopez, J. A., Marin-Martinez, F., Sanchez-Meca, J., Van den Noortgate, W., & Viechtbauer, W. (2014). Estimation of the predictive power of the model in mixed-effects meta-regression: A simulation study. British Journal of Mathematical and Statistical Psychology, 67(1), 30-48.

We only examined the standard mixed-effects meta-regression model with a single moderator, but found that the pseudo-R^2 statistic can be all over the place unless k is quite large.

Now you seem to have a larger number of estimates (170), but these are nested in 'only' 26 studies. So, I suspect that the estimate-level variance component is estimated fairly precisely, but not the study-level variance component. You may want to examine the profile plots (with the profile() function) and/or get (profile-likelihood) CIs of the variance components (using the confint() function). Probably the CI for the study-level variance component is quite wide.

Best,
Wolfgang

Wolfgang Viechtbauer, Ph.D., Statistician | Department of Psychiatry and    
Neuropsychology | Maastricht University | P.O. Box 616 (VIJV1) | 6200 MD    
Maastricht, The Netherlands | +31 (43) 388-4170 | http://www.wvbauer.com    

>-----Original Message-----
>From: R-help [mailto:r-help-bounces at r-project.org] On Behalf Of Duncan,
>Laura
>Sent: Monday, February 27, 2017 20:05
>To: r-help at r-project.org
>Subject: [R] Metafor multilevel metaregression: total variance increases
>when moderator added?
>
>Hi there,
>
>I am running a two level multilevel meta-regression of 170 estimates
>nested within 3 informants nested within 26 studies. I run the null model
>to get a pooled estimate with random effects at the informant level and
>study level.
>
>Then I test a series of potential moderators (one at a time, given small
>number of studies and adjust p-values for multiple testing). I use:
>(sum(Model1$sigma2) - sum(Model2$sigma2)) / sum(Model1$sigma2)
>to compute the proportional reduction in the total variance from here:
>http://stackoverflow.com/questions/22356450/getting-r-squared-from-a-
>mixed-effects-multilevel-model-in-metafor
>
>For one moderator, I get a negative value for reduced total variance and
>an unexpected negative coefficient. Based on Wolfgang's response in the
>link above this is possible "depending on the size of your dataset, those
>variance components may not be estimated very precisely and that can lead
>to such counter-intuitive results".
>
>I am trying to diagnose why this model is not being estimated properly and
>why I am getting an unexpected negative result. When I remove the second
>level from the model and run a single-level random effects models of 170
>estimates nested within 26 studies, the coefficient is positive and as we
>would expect.
>
>Does anyone have any suggestions for what might be going on or how I might
>diagnose the problem with this model?
>
>Thanks,
>Laura
>
>Laura Duncan, M.A.
>Research Coordinator
>Offord Centre for Child Studies
>McMaster University
>
>Tel: 905 525 9140 x21504
>Fax: 905 574 6665
>duncanlj at mcmaster.ca
>ontariochildhealthstudy.ca<www.ontariochildhealthstudy.ca>
>offordcentre.com
>
>Mailing Address                                               Courier
>Address
>1280 Main St. W. MIP 201A                           175 Longwood Rd. S.
>MIP 201A
>Hamilton, Ontario L8S 4K1                             Hamilton, Ontario
>L8P 0A1

Metafor multilevel metaregression: total variance increases when moderator added?

Thread (2 messages)