[R-meta] Computing Effect Size for Difference in Differences with Different Populations

Thu, Mar 16, 2023 10:36 AM

Hi Mika,

Depends on what you mean by 'best'. But note that escalc(measure="SMCRH, ...) computes (m1i-m2i)/sd1i, so you are using the post-treatment SD to standardize, which is a bit unusual and different from your second approach where you use the average pre-treatment SD to standardize. This aside, the two approaches should lead to rather similar estimates, especially if the pre-treatment SDs (assuming those are used in both approaches) are similar across the two groups.

Best,
Wolfgang

-----Original Message-----
From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces at r-project.org] On
Behalf Of Mika Manninen via R-sig-meta-analysis
Sent: Monday, 13 March, 2023 17:28
To: R meta
Cc: Mika Manninen
Subject: [R-meta] Computing Effect Size for Difference in Differences with
Different Populations

Dear community,

I am currently working on a meta-analysis that aims to examine the
difference in training effects between two populations. Both
populations underwent the same training, but at pre-test, the groups
have significantly different means and standard deviations (about
1-2sd difference in means).

I am interested in computing the effect size for the difference in
differences between the two groups. Specifically, I would like to know
what is the best way to calculate the effect size given the
significant difference in means and standard deviations at pre-test.

Would the below be roughly accurate (Option 1):

Option 1.

G1 <- escalc(measure="SMCRH", m1i=postm_G1, m2i=prem_G1,
sd1i=postsd_G1,ni=n_G1, sd2i = presd_G1, ri=c(rep(0.7,10)), data=G)
G2 <- escalc(measure="SMCRH", m1i=postm_G2, m2i=prem_G2,
sd1i=postsd_G2, ni=n_G2, sd2i = presd_G2, ri=c(rep(0.7,10)), data=G)
dat <- data.frame(yi = G1$yi - G2$yi, vi = G1$vi + G2$vi)

Option 2.

ES = (G1 post_mean - G2 pre_mean) - (G2 post_mean - G2 pre_mean) / pldpre_sd
pldpre_sd = sqrt((presdG1^2 + presdG2^2) / 2)

### dataset

set.seed(123)

postm_G1 <- rnorm(100, mean = 14, sd = 2.5)
prem_G1 <- rnorm(100, mean = 10, sd = 2)
postsd_G1 <- rnorm(100, mean = 1.4, sd = 0.2)
presd_G1 <- rnorm(100, mean = 1, sd = 0.2)
n_G1 <- rpois(100, lambda = 50)

postm_G2 <- rnorm(100, mean = 7.5, sd = 1.8)
prem_G2 <- rnorm(100, mean = 5, sd = 1.2)
postsd_G2 <- rnorm(100, mean = 0.9, sd = 0.2)
presd_G2 <- rnorm(100, mean = 0.6, sd = 0.2)
n_G2 <- rpois(100, lambda = 50)

G <- data.frame(postm_G1, prem_G1, postsd_G1, n_G1, presd_G1,
postm_G2, prem_G2, postsd_G2, n_G2, presd_G2)

###

Thank you in advance for your time and help.

Best wishes,
Mika

[R-meta] Computing Effect Size for Difference in Differences with Different Populations

Thread (5 messages)