[R-meta] Negative values of df in test of moderators using robust()
Hi Sebastian, I don't think it's possible to derive a clear rule-of-thumb for this because it depends on the configuration of the covariates (e.g., the number of studies contributing to each category, in the example I gave), not just on the total number of studies. Generally, Joshi et al. found that using cluster-wild bootstrap led to better Type-I error control and *higher power* for hypothesis tests involving multiple constraints (that is, hypothesis tests with 2 or more numerator degrees of freedom). Higher power is good, so it seems not unreasonable to use CWB routinely for such tests. More pragmatically (lazily?), I would definitely recommend using it whenever the denominator degrees of freedom of the regular robust test are small. James On Wed, Feb 1, 2023 at 12:52 AM R?hl, Sebastian <
sebastian.roehl at uni-tuebingen.de> wrote:
Dear James and Wolfgang, thank you so much for your answers! This is really helpful for me. Concerning the cluster wild bootstrapping: Do you have a rule of thumb below what number of studies and moderators it makes sense to use cluster wild bootstrapping? Best, Sebastian *Von:* James Pustejovsky <jepusto at gmail.com> *Gesendet:* Dienstag, 31. Januar 2023 22:08 *An:* Viechtbauer, Wolfgang (NP) < wolfgang.viechtbauer at maastrichtuniversity.nl> *Cc:* R?hl, Sebastian <sebastian.roehl at uni-tuebingen.de>; r-sig-meta-analysis at r-project.org *Betreff:* Re: Negative values of df in test of moderators using robust() The negative degrees of freedom arise because the small-sample approximation implemented in clubSandwich can become overly conservative when testing a hypothesis with large numerator degrees of freedom and a limited number of studies. For instance, suppose you are testing for differences in average effects between categories A, B, C, D, E, F, G, H, and I, so the numerator degrees of freedom will be 8 (A = B, A = C, A = D, etc.). If one (or more) of the categories has results from only two or three studies, then the denominator degrees of freedom can become negative and the test result should not be trusted. On the other hand, the QM test reported in the standard output is based on large-sample asymptotic approximations and should probably not be trusted either. In a recent simulation study by Megha Joshi ( https://www.jepusto.com/publication/cluster-wild-bootstrap-for-meta-analysis/), we found that using a cluster wild bootstrap test works much better in this situation. If you care about this particular test of moderators, I would recommend using this approach. It's implemented in the R package wildmeta: https://meghapsimatrix.github.io/wildmeta/ James On Tue, Jan 31, 2023 at 2:50 PM Viechtbauer, Wolfgang (NP) < wolfgang.viechtbauer at maastrichtuniversity.nl> wrote: Dear Sebastian, Yes, I assume that this is the issue. Here is a reproducible example to illustrate this: library(metafor) dat <- dat.konstantopoulos2011 res <- rma.mv(yi, vi, random = ~ 1 | district/school, data=dat, mods = ~ 0 + factor(year)) robust(res, cluster=district, clubSandwich=TRUE) CC-ing James, since this is really coming from clubSandwich. Best, Wolfgang
-----Original Message----- From: R-sig-meta-analysis [mailto:
r-sig-meta-analysis-bounces at r-project.org] On
Behalf Of R?hl, Sebastian Sent: Tuesday, 31 January, 2023 11:49 To: r-sig-meta-analysis at r-project.org Subject: [R-meta] Negative values of df in test of moderators using
robust()
Dear all, I have a question regarding robust(): I'm using robust with
clubSandwich-option
for testing moderator effects. In the test of moderators, a negative df2-value appears: Test of Moderators (coefficients 1:8): F(df1 = 8, df2 = -2.68) = 0.0000, p-val = NA In the standard output (without CRVE): Test of Moderators (coefficients 1:8): QM(df = 8) = 10.0880, p-val = 0.2589 How could I interpret this phenomen? Could this happen due to small
numbers of ES
for some of the moderators? Thank you for your help! Best, Sebastian **************************** Dr. Sebastian R?hl Eberhard Karls Universit?t T?bingen Institute for Educational Science T?bingen School of Education (T?SE) Wilhelmstra?e 31 / Room 302 D-72074 T?bingen Germany Phone: +49 7071 29-75527 Fax: +49 7071 29-35309 Email: sebastian.roehl at uni-tuebingen.de<mailto:
sebastian.roehl at uni-tuebingen.de>
Twitter: @sebastian_roehl @ResTeacherEdu