[R-meta] Categorical mixed effect models and interpretation of results
Here's a little bit of a different perspective on the need for species random effects in the model. I would argue that the question of whether or not to include species random effects in the model should be determined by the goal of the synthesis. On the one hand, if the goal is to draw generalizations across a population of species, and if it is reasonable to treat the 12 observed species as a sample from the population, then it would be appropriate to include random effects for species (or more generally, to start modeling the species effects using phylogenetic methods as Wolfgang suggested). On the other hand, if the 12 observed species constitute the full relevant population, or if inter-species variation is not of interest for the goals of the synthesis, or if one is content to restrict generalization to the observed species, then species random effects would not be appropriate. (Instead, one could simply treat species as a categorical covariate.) On Tue, Jun 19, 2018 at 8:55 AM Viechtbauer, Wolfgang (SP) <
wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:
Hi Alex, Please keep the mailing list in cc. I would not say that the results are necessarily 'incorrect'. However, just adding random effects for study assumes that there is no heterogeneity in the outcomes for different experiments within studies, which is a big assumption to make. Also, not adding random effects for species assumes that outcomes for the same species are no more similar to each other than they are when outcomes are measured in different species. Again, that's a questionable assumption. In fact, outcomes in different species may be correlated to the degree that species resemble each other, so that gets us into phylogenetic meta-analyses. See, for example: Nakagawa, S., & Santos, E. S. A. (2012). Methodological issues and advances in biological meta-analysis. Evolutionary Ecology, 26(5), 1253-1274. Noble, D. W., Lagisz, M., O'Dea R, E., & Nakagawa, S. (2017). Non-independence and sensitivity analyses in ecological and evolutionary meta-analyses. Molecular Ecology, 26(9), 2410-2425. Ignoring such issues could lead to underestimation of the uncertainty in parameter estimates and hence inflated Type I error rates. Best, Wolfgang -----Original Message----- From: Alexander Sullivan (BIO - Student) [mailto: Alexander.J.Sullivan at uea.ac.uk] Sent: Tuesday, 19 June, 2018 12:32 To: Viechtbauer, Wolfgang (SP) Subject: Re: Categorical mixed effect models and interpretation of results Dear Wolfgang, Thank you for your prompt reply. This does appear to improve my model. However, I have noticed in the meta-analytical literature especially within my research area, ocean acidification, that many authors simply use study id as their random effect. Does this mean their results are incorrect? or simply less precise than a more complicated model? Best, Alex MSc Ecology and Conservation student University of East Anglia, UK
________________________________________ From: Viechtbauer, Wolfgang (SP) < wolfgang.viechtbauer at maastrichtuniversity.nl> Sent: 19 June 2018 09:31:52 To: Alexander Sullivan (BIO - Student); r-sig-meta-analysis at r-project.org Subject: RE: Categorical mixed effect models and interpretation of results Hi Alex, Yes, in principle this is right. If 'Treatment' only has two levels, then the QM-test and the test of the treatment coefficient are identical, so you can also just look at the latter. However, I think your random effects structure is too simple given what you wrote. At the least, it should be something like: random = ~ 1 | Study_id / Exp_id where Exp_id is, as the name implies, the experiment id. See: http://www.metafor-project.org/doku.php/analyses:konstantopoulos2011 and esp. the "A Common Mistake in the Three-Level Model" section. You might also want to consider adding random effects for species. For example: random = list(~ 1 | Study_id / Exp_id, ~ 1 | Species) would add species random effects (as a crossed random effect, not nested within study and/or experiment). Best, Wolfgang -----Original Message----- From: R-sig-meta-analysis [mailto: r-sig-meta-analysis-bounces at r-project.org] On Behalf Of Alexander Sullivan (BIO - Student) Sent: Tuesday, 19 June, 2018 10:58 To: r-sig-meta-analysis at r-project.org Subject: [R-meta] Categorical mixed effect models and interpretation of results Dear all, In my meta-analysis I have [say] 100 experiments, from 30 studies, across 12 species. These experiments can be equally divided into two treatment levels [low and high]. If I want to determine if there is a significant difference in effect sizes between these two treatment levels would this be the right code: res <- rma.mv(yi, vi, method = "REML", data = mydata, mods = ~factor(Treatment), random = ~1|Study_id) ... and then from the output of this model if the Qm statistic is significant I could say there is a significant difference in effect sizes between the two treatments? Thank you for your time, Alex MSc Ecology and Conservation student University of East Anglia, UK _______________________________________________ R-sig-meta-analysis mailing list R-sig-meta-analysis at r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis