[R-meta] Fwd: Overlapping CIs with significant difference among subgroups

-----Original Message-----
From: Rafael Rios Moura [mailto:biorafaelrm at gmail.com]
Sent: Tuesday, 06 October, 2020 20:25
To: r-sig-meta-analysis at r-project.org; Viechtbauer, Wolfgang (SP)
Subject: Fwd: Overlapping CIs with significant difference among subgroups

ATTACHMENT(S) REMOVED: dataset.csv | pruned_super-tree.tre | script.R

Dear Wolfgang and All,

Few months ago, I sent this email about a result obtained from a mixed
effects MLMA, controlling for phylogenetic non-independence. I tested the
difference between two levels of a moderator and obtained two close means
(0.39 and 0.31) with highly overlapping CIs. However, the omnibus test
detected a difference between estimates. Could it be a problem with my code
or the test? Or am I not using the "predict" function correctly? My dataset
and script are attached. I am grateful for contributions.

Best wishes,

Dear Rafael,

The SEs of the predicted average outcomes for the various levels can be
quite different than the SEs of the difference between levels.

You can get the predicted average outcome for all four levels with:

pred.r = predict(res, transf=transf.ztor, digits=3, newmods =
rbind(c(0,0,0), c(1,0,0), c(0,1,0), c(1,1,1)))
pred.r

And then you can plot them with:

forest(pred.r$pred, ci.lb=pred.r$ci.lb, ci.ub=pred.r$ci.ub,
slab=c("Without grouping","Temporal grouping","Spatial grouping", "Both"))

Indeed, CIs are wide and overlap. But let's now compute the predicted
average difference between levels when compared against the "Without
grouping" level:

pred.r = predict(res, transf=transf.ztor, digits=3, newmods =
rbind(c(1,0,0) - c(0,0,0), c(0,1,0) - c(0,0,0), c(1,1,1) - c(0,0,0)),
intercept=FALSE)
pred.r

forest(pred.r$pred, ci.lb=pred.r$ci.lb, ci.ub=pred.r$ci.ub, slab=c("Diff
Temporal grouping","Diff Spatial grouping", "Diff Both"))

Now the CIs are quite narrow and exlude 0.

This aside, I would recommend that you include random effects for species
twice, once without and once with the phylogenetic correlation matrix:

h$speciesIDnon <- h$speciesID
res <- rma.mv(zf, vzf, mods=~sce_temporal*sce_spatial, random = list
(~1|effectsizeID, ~1|studyID, ~1|speciesIDnon, ~1|speciesID),
R=list(speciesID=corr), data=h)

This is model (15) from:

Nakagawa, S., & Santos, E. S. A. (2012). Methodological issues and
advances in biological meta-analysis. Evolutionary Ecology, 26(5),
1253-1274.

Conclusions do not change as far as I can tell, but I would still go with
that model. A LRT also shows that this model fits significantly better:

res0 <- rma.mv(zf, vzf, mods=~sce_temporal*sce_spatial, random = list
(~1|effectsizeID, ~1|studyID, ~1|speciesID), R=list(speciesID=corr), data=h)
anova(res, res0)

Best,
Wolfgang

-----Original Message-----
From: Rafael Rios Moura [mailto:biorafaelrm at gmail.com]
Sent: Tuesday, 06 October, 2020 20:25
To: r-sig-meta-analysis at r-project.org; Viechtbauer, Wolfgang (SP)
Subject: Fwd: Overlapping CIs with significant difference among subgroups

ATTACHMENT(S) REMOVED: dataset.csv | pruned_super-tree.tre | script.R

Dear Wolfgang and All,

Few months ago, I sent this email about a result obtained from a mixed
effects MLMA, controlling for phylogenetic non-independence. I tested the
difference between two levels of a moderator and obtained two close means
(0.39 and 0.31) with highly overlapping CIs. However, the omnibus test
detected a difference between estimates. Could it be a problem with my

code

or the test? Or am I not using the "predict" function correctly? My

dataset

and script are attached. I am grateful for contributions.

Best wishes,