[R-meta] Appending a risk-of-bias traffic-light plot to a 'three-level' forest plot

I would suggest to use test="t" with dfs="contain". There is a bit of discussion here:

https://wviechtb.github.io/metafor/reference/rma.mv.html#tests-and-confidence-intervals

Roughly, z-tests tend to be too liberal, which we can counteract by using t-tests. But the default df calculation is quite simplistic and the "contain" method is an improvement on that. Still far from perfect, but a bit better.

Best,
Wolfgang

-----Original Message-----
From: Joshua Bernal [mailto:jdkb9701 at connect.hku.hk]
Sent: Friday, 18 February, 2022 17:23
To: Viechtbauer, Wolfgang (SP); r-sig-meta-analysis at r-project.org
Subject: Re: [R-meta] Appending a risk-of-bias traffic-light plot to a 'three-
level' forest plot

Dear Dr. Viechtbauer,

Thanks very much for your timely help! I'm delighted to learn about a
simple alternative approach for creating the traffic light part. I
would like to follow-up on the first part of the query on rma.mv /
aggregated data...

As the help page of the rma.mv function mentions: one can set
dfs="contain" (which automatically also sets test="t")

I checked to see whether the results would be the same as test="t" if
I specify dfs="contain", and whether specifying test="t" with versus
without dfs="contain" would yield the same results.

# dfs="contain"
res <- rma.mv(yi, vi, random = ~ 1 | author/outcome, data = dat, dfs =
"contain", method = "REML", slab = author)

estimate      se    zval    pval   ci.lb   ci.ub
 0.2495  0.0738  3.3828  0.0007  0.1049  0.3940  ***

# test="t"
res <- rma.mv(yi, vi, random = ~ 1 | author/outcome, data = dat, test
= "t", method = "REML", slab = author)

estimate      se    tval  df    pval   ci.lb   ci.ub
 0.2495  0.0738  3.3828  15  0.0041  0.0923  0.4067  **

# test="t" and dfs="contain"
res <- rma.mv(yi, vi, random = ~ 1 | author/outcome, data = dat, test
= "t", dfs = "contain", method = "REML", slab = author)

estimate      se    tval  df    pval   ci.lb   ci.ub
 0.2495  0.0738  3.3828   5  0.0196  0.0599  0.4391  *

I was wondering why the results differ, and importantly when would it
be appropriate or sensible to use each of these approaches?
Ultimately, I would like to know how I should determine the
appropriate method for this part of the meta-analysis and generally
what to consider when doing so (e.g., study sample size, number of
studies, number of effect estimates per study)? For example, is it
acceptable to use test="z" for a meta-analysis of eight studies with
sample sizes of 66, 50, 38, 23, 23, 18, 12, 10 versus five studies
with sample sizes of 50, 35, 28, 12, 10; or is it more sensible to be
'conservative' and use test="t" in either one (or both) cases?

Best regards,
Josh