[R-meta] Additional Info: Pairwise moderator testing in multilevel meta-analysis with CRVE / CIs

Hi all,

Just an addition to my question from yesterday:
Additionally to using the robust() and anova() function, I also tried 
out
Wald_test() from the clubSandwich packacke.
The results are the same (with F instead of T statistics):

Wald_test(out_3, constraints = constrain_pairwise(1:3), vcov="CR2")

$`out_acad - out_intg`
 test Fstat df_num df_denom  p_val sig
  HTZ  4.14      1     10.9 0.0669   .

$`out_socem - out_intg`
 test Fstat df_num df_denom p_val sig
  HTZ 0.225      1     13.2 0.643

$`out_socem - out_acad`
 test Fstat df_num df_denom   p_val sig
  HTZ  18.7      1      9.6 0.00165  **

Can anybody help me?

Thank you.

All the best,
Sebastian

-----Urspr?ngliche Nachricht-----
Von: R-sig-meta-analysis <r-sig-meta-analysis-bounces at r-project.org> 
Im Auftrag von R?hl, Sebastian via R-sig-meta-analysis
Gesendet: Donnerstag, 9. Februar 2023 12:30
An: r-sig-meta-analysis at r-project.org
Cc: R?hl, Sebastian <sebastian.roehl at uni-tuebingen.de>
Betreff: [R-meta] Pairwise moderator testing in multilevel 
meta-analysis with CRVE / CIs

Hi,

I have the following problem:
I am conducting a multilevel meta-analysis using metafor with cluster 
robust variance estimation and want to test the moderating effect of 
different kinds of outcomes. Additionally I want to test whether the 
several outcomes differ significantly from each other.
Here is an example:
out_3 <- rma.mv(zr, V=var, random = ~ 1| Sample_ID / number, mods = ~ 
-1
+ out_intg + out_acad + out_socem,
                       data = data_int) out_3_rob <- robust(out_3, 
Sample_ID, clubSandwich = T) anova(out_3_rob, 
X=rbind(c(-1,1,0),c(-1,0,1), c(0,-1,1)))

The robust model result shows C.I. that overlap.
Model Results:

           estimate     se?   tval?    df?   pval?  ci.lb?  ci.ub?     ?
out_intg     0.2302  0.0231  9.9484  30.84  <.0001  0.1830  0.2773  ***
out_acad     0.1646  0.0220  7.4677  17.36  <.0001  0.1182  0.2111  ***
out_socem    0.2458  0.0278  8.8510  22.27  <.0001  0.1882  0.3034  ***

BUT the anova results show significant differences between 2 outcomes:

Hypotheses:

1:  -out_intg + out_acad = 0

2: -out_intg + out_socem = 0

3: -out_acad + out_socem = 0



Results:

   estimate     se    tval    df   pval

1:  -0.0655 0.0322 -2.0349 10.92 0.0669

2:   0.0157 0.0330  0.4742 13.21 0.6431

3:   0.0812 0.0188  4.3264  9.60 0.0016

Do I have a thinking error here about the ANOVA or is this pairwise 
testing not possible with the CRVE-results?
Perhaps I am also interpreting the C.I.s incorrectly? If I calculate a 
pairwise comparison with the non-robust model, I also get significant 
difference although also the non-robust C.I. overlap.

Thank you very much for your help!
Best,
Sebastian

****************************
Dr. Sebastian R?hl
Eberhard Karls Universit?t T?bingen
Institute for Educational Science
T?bingen School of Education (T?SE)
Wilhelmstra?e 31 / Room 302
D-72074 T?bingen
Germany

Phone: +49 7071 29-75527
Fax: +49 7071 29-35309
Email: sebastian.roehl at uni-tuebingen.de<mailto:
sebastian.roehl at uni-tuebingen.de>
Twitter: @sebastian_roehl  @ResTeacherEdu



        [[alternative HTML version deleted]]