[R-meta] Questions about Omnibus tests
Dear Rafael I will let Wolfgang respond about the details of models but as far as heterogeneity is concerned. Eliminating outliers without concrete reasons may be removing the most interesting observations. If the primary studies are well estimated then heterogeneity is bound to be large. See for example R?cker, G, Schwarzer, G, Carpenter, J R and Schumacher, M, Undue reliance on I^2 in assessing heterogeneity may mislead BMC Medical Research Methodology 2008 (8) 79 It is really the pattern of heterogeneity which you need to examine in your funnel plot and then try to explain it. Michael
On 03/11/2018 23:31, Rafael Rios wrote:
Dear Wolfgang, Could you please help me again with new questions? Should I build model1 rather than model2 to control for the dependency among studyID and effectsizeID? model1=rma.mv <http://rma.mv>(zf, vzf, mods=~mate_choice, random=list(~1|studyID/effectsizeID, ~1|species1), data = h_mc) model2=rma.mv <http://rma.mv>(zf, vzf, mods=~mate_choice, random=list(~1|effectsizeID, ~1|studyID, ~1|species1), data = h_mc) I used your script to calculate I? and found a high heterogeneity in my model (86.63%). #I?: http://www.metafor-project.org/doku.php/tips:i2_multilevel_multivariate W <- diag(1/h_mc$vzf) X <- model.matrix(model1) P <- W - W %*% X %*% solve(t(X) %*% W %*% X) %*% t(X) %*% W 100 * sum(meta$sigma2) / (sum(meta$sigma2) + (meta$k-meta$p)/sum(diag(P))) Do you have suggestions on how to handle with high heterogeneity among effect sizes? How may I conduct sensitivity tests in a multilevel meta-analysis using metafor? I identified (using a funnel plot) and removed outliers to reduce the heterogeneity and redo the model. Is this approach suitable to evaluate potential bias in results? Or are there better alternatives? Best wishes, Rafael.
__________________________________________________________ Dr. Rafael Rios Moura /scientia amabilis/ Behavioral Ecologist, PhD Postdoctoral Researcher Universidade Estadual de Campinas (UNICAMP) Campinas, S?o Paulo, Brazil Curr?culo Lattes: http://lattes.cnpq.br/4264357546465157 ORCID: http://orcid.org/0000-0002-7911-4734 Research Gate: https://www.researchgate.net/profile/Rafael_Rios_Moura2 <http://buscatextual.cnpq.br/buscatextual/visualizacv.do?id=K4244908A8> Em ter, 30 de out de 2018 ?s 16:01, Rafael Rios <biorafaelrm at gmail.com <mailto:biorafaelrm at gmail.com>> escreveu: Dear Wolfgang, Thank you for the amazing clarifications! I think I finally have a better picture about the meta-analytic procedures. Best wishes, Rafael. __________________________________________________________ Dr. Rafael Rios Moura /scientia amabilis/ Behavioral Ecologist, PhD Postdoctoral Researcher Universidade Estadual de Campinas (UNICAMP) Campinas, S?o Paulo, Brazil Curr?culo Lattes: http://lattes.cnpq.br/4264357546465157 ORCID: http://orcid.org/0000-0002-7911-4734 Research Gate: https://www.researchgate.net/profile/Rafael_Rios_Moura2 <http://buscatextual.cnpq.br/buscatextual/visualizacv.do?id=K4244908A8> Em ter, 30 de out de 2018 ?s 15:28, Viechtbauer, Wolfgang (SP) <wolfgang.viechtbauer at maastrichtuniversity.nl <mailto:wolfgang.viechtbauer at maastrichtuniversity.nl>> escreveu: Dear Rafael, 1. "Does the QM-test, with an intercept in the model, evaluates if the average true outcomes of subgroups differ from the reference level or from 0?" >From the reference level. "Why are the graph results so different from the QM-test with an intercept in the model?" Your graph is not correct. It should be: preds <- predict(meta, newmods=rbind(c(0,0), c(1,0), c(0,1))) forest(preds$pred, sei=preds$se, slab=c("female", "male", "mutual")) The differences between the three levels are small. "Should I evaluate results using anova(meta,btt=1:3)?" anova(meta,btt=1:3) tests if all 3 groups have a zero effect. That does not test for differences between groups. "Was the argument linfct=rbind(c(0,0,1)) used to compare the subgroups of female choice (reference level) and male choice?" No, this compares 'mutual' with 'female'. "What am I evaluating by using summary(glht(meta, linfct=rbind(female=c(1,0,0), male=c(0,1,0))), test=Chisqtest())" You are evaluating whether the intercept (and hence the effect for 'female') is 0 and whether there is a difference between 'male' and 'female'. 2. "What is the best approach to measure heterogeneity in a multilevel meta-analysis?" I don't know what is best. The link you posted provides some possibilities for computing I^2-like measures for multilevel/multivariate models. 3. "I used the standard deviation to weight the effect sizes, according to Zaykin (2011). Is variance a better measure of weight than se in a multilevel meta-analysis?" As mentioned by Michael, this article is irrelevant. 4. "An alternative could be to include this potential_sce as a fixed variable." Sure. "Is this model more appropriate?: meta=rma.mv <http://rma.mv>(zf, sezf, mods=~mate_choice+potential_sce, random = list (~1|effectsizeID, ~1|studyID, ~1|species1), data = h_mc)" You should pass the variances to the function: meta=rma.mv <http://rma.mv>(zf, vzf, mods=~mate_choice+potential_sce, random = list (~1|effectsizeID, ~1|studyID, ~1|species1), data = h_mc) Best, Wolfgang -----Original Message----- From: Rafael Rios [mailto:biorafaelrm at gmail.com <mailto:biorafaelrm at gmail.com>] Sent: Tuesday, 30 October, 2018 6:16 To: Viechtbauer, Wolfgang (SP) Cc: Michael Dewey; r-sig-meta-analysis at r-project.org <mailto:r-sig-meta-analysis at r-project.org> Subject: Re: [R-meta] Questions about Omnibus tests Dear Wolfgang, Thank you for the very helpful advices! I will be grateful if you could help me again with my new questions. I organized them in the topics bellow. 1. Does the QM-test, with an intercept in the model, evaluates if the average true outcomes of subgroups differ from the reference level or from 0? I found a p>0.05, probably meaning that there is no difference among subgroups. However, if you analyze the graph, there a higher effect size for the subgroup of female choice compared to others. So, I am not sure about the best approach to evaluate differences among outcomes. Why are the graph results so different from the QM-test with an intercept in the model? Should I evaluate results using anova(meta,btt=1:3)? You also suggested that the script for pairwise comparisons was wrong. According to the link that you provided, it can also be drawn as?summary(glht(meta, linfct=rbind(c(0,0,1), c(0,1,0), c(0,-1,1))), test=adjusted("none")). Was the argument linfct=rbind(c(0,0,1)) used to compare the subgroups of female choice (reference level) and male choice? What am I evaluating by using summary(glht(meta, linfct=rbind(female=c(1,0,0), male=c(0,1,0))), test=Chisqtest())? 2. Thank you for the correction of I? formula. What is the best approach to measure heterogeneity in a multilevel meta-analysis? Maybe, this one: http://www.metafor-project.org/doku.php/tips:i2_multilevel_multivariate 3. I used the standard deviation to weight the effect sizes, according to Zaykin (2011). Is variance a better measure of weight than se in a multilevel meta-analysis? Reference: D. V. Zaykin, Optimally weighted Z-test is a powerful method for combining probabilities in meta-analysis. J. Evol. Biol. 24, 1836?1841 (2011). 4. Finally, I agree with the exclusion of potential_sce as a random variable. However, I need to control for this variable. An alternative could be to include this potential_sce as a fixed variable. Is this model more appropriate?: meta=rma.mv <http://rma.mv>(zf, sezf, mods=~mate_choice+potential_sce, random = list (~1|effectsizeID, ~1|studyID, ~1|species1), data = h_mc). Thank you again for the help. Best wishes, Rafael. __________________________________________________________ Dr. Rafael Rios Moura scientia amabilis Behavioral Ecologist,?PhD Postdoctoral Researcher Universidade Estadual de Campinas (UNICAMP) Campinas, S?o Paulo, Brazil Curr?culo Lattes: http://lattes.cnpq.br/4264357546465157 ORCID: http://orcid.org/0000-0002-7911-4734 Research Gate: https://www.researchgate.net/profile/Rafael_Rios_Moura2