Lukasz Stasielowicz
Osnabr?ck University
Institute for Psychology
Research methods, psychological assessment, and evaluation
Seminarstra?e 20
49074 Osnabr?ck (Germany)
Am 07.04.2022 um 12:00 schrieb r-sig-meta-analysis-request at r-project.org:
> Send R-sig-meta-analysis mailing list submissions to
> r-sig-meta-analysis at r-project.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
> or, via email, send a message with subject or body 'help' to
> r-sig-meta-analysis-request at r-project.org
>
> You can reach the person managing the list at
> r-sig-meta-analysis-owner at r-project.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of R-sig-meta-analysis digest..."
>
>
> Today's Topics:
>
> 1. Moderator/predictor in multi-level meta-analysis (Hanel, Paul H P)
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Wed, 6 Apr 2022 22:51:11 +0000
> From: "Hanel, Paul H P" <p.hanel at essex.ac.uk>
> To: "r-sig-meta-analysis at r-project.org"
> <r-sig-meta-analysis at r-project.org>
> Subject: [R-meta] Moderator/predictor in multi-level meta-analysis
> Message-ID:
> <DB6PR0601MB23578B5ADC31A20006821A4DAEE79 at DB6PR0601MB2357.eurprd06.prod.outlook.com>
>
> Content-Type: text/plain; charset="utf-8"
>
> Hello,
>
> When running a standard random effects meta-analysis with one categorical dichotomous predictor/moderator, the estimate of the moderator is very similar to the difference that you get when running a random effects meta-analysis for each level of the predictor separately. This is however different to when running a multi-level meta-analysis.
>
> I am running a meta-analysis with over 300 effect sizes in which all studies have a control and a treatment group. The mean differences between the control and the intervention group are quantified with Hedges' gs. One moderator/predictor of these differences is sample type: students vs non-students. When using a standard random-effects model we get:
> dfm <- subset(df, sample_type == 0) # sample_type has two levels: students (coded as 0) vs other (coded as 1 in sample_type variable)
> dft <- subset(df, sample_type == 1)
> rma(yi, vi, data = dfm) # gives us g = .3855. yi are Hedges' gs.
> rma(yi, vi, data = dft) # gives us g = .59.
> rma(yi, vi, mods = sample_type, data=df) # Now full dataset with sample_type as categorical moderator/predictor. Estimate = .21, which is very close to .59 - .3855 = .20. Also, the intercept is very close sample_type = 0, intercept = .3861.
>
> However, when switching to a multi-level model with effect sizes nested in studies nested in papers, things get more complicated. Using the same datasets as above we get
> rma.mv(yi, vi, random = list(~ 1 | effectID, ~ 1 | StudyID, ~ 1 | PaperID), tdist = TRUE, data=dfm) # gives us g = .45, 95%-CI [.33, .59]
> rma.mv(yi, vi, random = list(~ 1 | effectID, ~ 1 | StudyID, ~ 1 | PaperID), tdist = TRUE, data=dft) # gives us g = .61, 95%-CI [.48, .73]
> rma.mv(yi, vi, mods = ~ sample_type, random = list(~ 1 | effectID, ~ 1 | StudyID, ~ 1 | PaperID), tdist = TRUE, data=df) # Now full dataset with sample_type as categorical moderator/predictor. Estimate = .08, 95%-CI[-.07, .24] which is quite a bit off from .61 - .45 = .16. Interestingly, the intercept is also off from sample_type = 0. Intercept = .48, even though I assumed it would be .45.
>
> In short, why is the estimate for the multi-level moderator test not similar to the difference between the two effect sizes obtained separately for each of the two subgroups - assuming there is no error in the code above? When looking at the confidence intervals, the difference between students vs others should be significant, but including the sample_type as moderator does not yield a significant finding.
>
> I got similar discrepancies when using a different moderator.
>
> Thank you,
> Paul
>
>
> [[alternative HTML version deleted]]
>
>
>
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> R-sig-meta-analysis mailing list @ R-sig-meta-analysis at r-project.org
> To manage your subscription to this mailing list, go to:
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>
>
> ------------------------------
>
> End of R-sig-meta-analysis Digest, Vol 59, Issue 2
> **************************************************