[R-meta] Negative values of df in test of moderators using robust()
Dear James and Wolfgang, thank you so much for your answers! This is really helpful for me. Concerning the cluster wild bootstrapping: Do you have a rule of thumb below what number of studies and moderators it makes sense to use cluster wild bootstrapping? Best, Sebastian Von: James Pustejovsky <jepusto at gmail.com> Gesendet: Dienstag, 31. Januar 2023 22:08 An: Viechtbauer, Wolfgang (NP) <wolfgang.viechtbauer at maastrichtuniversity.nl> Cc: R?hl, Sebastian <sebastian.roehl at uni-tuebingen.de>; r-sig-meta-analysis at r-project.org Betreff: Re: Negative values of df in test of moderators using robust() The negative degrees of freedom arise because the small-sample approximation implemented in clubSandwich can become overly conservative when testing a hypothesis with large numerator degrees of freedom and a limited number of studies. For instance, suppose you are testing for differences in average effects between categories A, B, C, D, E, F, G, H, and I, so the numerator degrees of freedom will be 8 (A = B, A = C, A = D, etc.). If one (or more) of the categories has results from only two or three studies, then the denominator degrees of freedom can become negative and the test result should not be trusted. On the other hand, the QM test reported in the standard output is based on large-sample asymptotic approximations and should probably not be trusted either. In a recent simulation study by Megha Joshi (https://www.jepusto.com/publication/cluster-wild-bootstrap-for-meta-analysis/), we found that using a cluster wild bootstrap test works much better in this situation. If you care about this particular test of moderators, I would recommend using this approach. It's implemented in the R package wildmeta: https://meghapsimatrix.github.io/wildmeta/ James
On Tue, Jan 31, 2023 at 2:50 PM Viechtbauer, Wolfgang (NP) <wolfgang.viechtbauer at maastrichtuniversity.nl<mailto:wolfgang.viechtbauer at maastrichtuniversity.nl>> wrote:
Dear Sebastian, Yes, I assume that this is the issue. Here is a reproducible example to illustrate this: library(metafor) dat <- dat.konstantopoulos2011 res <- rma.mv<http://rma.mv>(yi, vi, random = ~ 1 | district/school, data=dat, mods = ~ 0 + factor(year)) robust(res, cluster=district, clubSandwich=TRUE) CC-ing James, since this is really coming from clubSandwich. Best, Wolfgang
-----Original Message----- From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces at r-project.org<mailto:r-sig-meta-analysis-bounces at r-project.org>] On Behalf Of R?hl, Sebastian Sent: Tuesday, 31 January, 2023 11:49 To: r-sig-meta-analysis at r-project.org<mailto:r-sig-meta-analysis at r-project.org> Subject: [R-meta] Negative values of df in test of moderators using robust() Dear all, I have a question regarding robust(): I'm using robust with clubSandwich-option for testing moderator effects. In the test of moderators, a negative df2-value appears: Test of Moderators (coefficients 1:8): F(df1 = 8, df2 = -2.68) = 0.0000, p-val = NA In the standard output (without CRVE): Test of Moderators (coefficients 1:8): QM(df = 8) = 10.0880, p-val = 0.2589 How could I interpret this phenomen? Could this happen due to small numbers of ES for some of the moderators? Thank you for your help! Best, Sebastian **************************** Dr. Sebastian R?hl Eberhard Karls Universit?t T?bingen Institute for Educational Science T?bingen School of Education (T?SE) Wilhelmstra?e 31 / Room 302 D-72074 T?bingen Germany Phone: +49 7071 29-75527 Fax: +49 7071 29-35309 Email: sebastian.roehl at uni-tuebingen.de<mailto:sebastian.roehl at uni-tuebingen.de><mailto:sebastian.roehl at uni-tuebingen.de<mailto:sebastian.roehl at uni-tuebingen.de>> Twitter: @sebastian_roehl @ResTeacherEdu