Message-ID: <bc1bd0109e734b33b7e4f181fd455d36@uni-tuebingen.de>
Date: 2023-02-10T16:11:48Z
From: Röhl, Sebastian
Subject: [R-meta] Additional Info: Pairwise moderator testing in multilevel meta-analysis with CRVE / CIs
In-Reply-To: <CAFUVuJzU7vG8K7o6qUr+Ls0vsEXthTE+cmNb7ogv4522kp7hnA@mail.gmail.com>
Hi James,
thank you very much! This helps me a lot!
Best regards,
Sebastian
-----Urspr?ngliche Nachricht-----
Von: R-sig-meta-analysis <r-sig-meta-analysis-bounces at r-project.org> Im Auftrag von James Pustejovsky via R-sig-meta-analysis
Gesendet: Freitag, 10. Februar 2023 15:12
An: R Special Interest Group for Meta-Analysis <r-sig-meta-analysis at r-project.org>
Cc: James Pustejovsky <jepusto at gmail.com>
Betreff: Re: [R-meta] Additional Info: Pairwise moderator testing in multilevel meta-analysis with CRVE / CIs
Hi Sebastian,
Pairwise tests are definitely possible when using CRVE. The issue is that overlap of confidence intervals is not generally a valid method for gauging statistical significance of differences.
When comparing the means of *independent* samples, confidence interval overlap is conservative, so overlap does not imply statistical non-significance of differences in means. See Schenker & Gentleman (2001; https://doi.org/10.1198/000313001317097960), Austin & Hux (2002;
https://doi.org/10.1067/mva.2002.125015) or many others.
If the means are from *dependent* samples (as could be the case for your meta-regression results), there is no direct correspondence between CI overlap and statistical significance. This is because the SE for the difference in means depends not just on the SEs for the means but also on the sampling covariance between them. As a simple example, consider the confidence intervals for the means of A and B, based on a sample of N = 100 from a bivariate normal distribution where meanB = meanA + 0.1, sdA = sdB = 1, and cor(A,B) = 0.9. The confidence intervals will have a probability of overlapping but the difference in means will be fairly precisely estimated because the correlation is so high.
James
On Fri, Feb 10, 2023 at 2:50 AM R?hl, Sebastian via R-sig-meta-analysis < r-sig-meta-analysis at r-project.org> wrote:
> Hi all,
>
> Just an addition to my question from yesterday:
> Additionally to using the robust() and anova() function, I also tried
> out
> Wald_test() from the clubSandwich packacke.
> The results are the same (with F instead of T statistics):
> > Wald_test(out_3, constraints = constrain_pairwise(1:3), vcov="CR2")
> $`out_acad - out_intg`
> test Fstat df_num df_denom p_val sig
> HTZ 4.14 1 10.9 0.0669 .
>
> $`out_socem - out_intg`
> test Fstat df_num df_denom p_val sig
> HTZ 0.225 1 13.2 0.643
>
> $`out_socem - out_acad`
> test Fstat df_num df_denom p_val sig
> HTZ 18.7 1 9.6 0.00165 **
>
> Can anybody help me?
>
> Thank you.
>
> All the best,
> Sebastian
>
> -----Urspr?ngliche Nachricht-----
> Von: R-sig-meta-analysis <r-sig-meta-analysis-bounces at r-project.org>
> Im Auftrag von R?hl, Sebastian via R-sig-meta-analysis
> Gesendet: Donnerstag, 9. Februar 2023 12:30
> An: r-sig-meta-analysis at r-project.org
> Cc: R?hl, Sebastian <sebastian.roehl at uni-tuebingen.de>
> Betreff: [R-meta] Pairwise moderator testing in multilevel
> meta-analysis with CRVE / CIs
>
> Hi,
>
> I have the following problem:
> I am conducting a multilevel meta-analysis using metafor with cluster
> robust variance estimation and want to test the moderating effect of
> different kinds of outcomes. Additionally I want to test whether the
> several outcomes differ significantly from each other.
> Here is an example:
> out_3 <- rma.mv(zr, V=var, random = ~ 1| Sample_ID / number, mods = ~
> -1
> + out_intg + out_acad + out_socem,
> data = data_int) out_3_rob <- robust(out_3,
> Sample_ID, clubSandwich = T) anova(out_3_rob,
> X=rbind(c(-1,1,0),c(-1,0,1), c(0,-1,1)))
>
> The robust model result shows C.I. that overlap.
> Model Results:
>
> estimate se? tval? df? pval? ci.lb? ci.ub? ?
> out_intg 0.2302 0.0231 9.9484 30.84 <.0001 0.1830 0.2773 ***
> out_acad 0.1646 0.0220 7.4677 17.36 <.0001 0.1182 0.2111 ***
> out_socem 0.2458 0.0278 8.8510 22.27 <.0001 0.1882 0.3034 ***
>
> BUT the anova results show significant differences between 2 outcomes:
>
> Hypotheses:
>
> 1: -out_intg + out_acad = 0
>
> 2: -out_intg + out_socem = 0
>
> 3: -out_acad + out_socem = 0
>
>
>
> Results:
>
> estimate se tval df pval
>
> 1: -0.0655 0.0322 -2.0349 10.92 0.0669
>
> 2: 0.0157 0.0330 0.4742 13.21 0.6431
>
> 3: 0.0812 0.0188 4.3264 9.60 0.0016
>
> Do I have a thinking error here about the ANOVA or is this pairwise
> testing not possible with the CRVE-results?
> Perhaps I am also interpreting the C.I.s incorrectly? If I calculate a
> pairwise comparison with the non-robust model, I also get significant
> difference although also the non-robust C.I. overlap.
>
> Thank you very much for your help!
> Best,
> Sebastian
>
> ****************************
> Dr. Sebastian R?hl
> Eberhard Karls Universit?t T?bingen
> Institute for Educational Science
> T?bingen School of Education (T?SE)
> Wilhelmstra?e 31 / Room 302
> D-72074 T?bingen
> Germany
>
> Phone: +49 7071 29-75527
> Fax: +49 7071 29-35309
> Email: sebastian.roehl at uni-tuebingen.de<mailto:
> sebastian.roehl at uni-tuebingen.de>
> Twitter: @sebastian_roehl @ResTeacherEdu
>
>
>
> [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-meta-analysis mailing list @ R-sig-meta-analysis at r-project.org
> To manage your subscription to this mailing list, go to:
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
> _______________________________________________
> R-sig-meta-analysis mailing list @ R-sig-meta-analysis at r-project.org
> To manage your subscription to this mailing list, go to:
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>
[[alternative HTML version deleted]]
_______________________________________________
R-sig-meta-analysis mailing list @ R-sig-meta-analysis at r-project.org To manage your subscription to this mailing list, go to:
https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis