[R-meta] Computing Effect Size for Difference in Differences with Different Populations

Thu, Mar 16, 2023 3:46 PM

Hey Wolfgang,

Thank you very much for the reply.

My mistake with the sd1i argument, I wasn't supposed to be using the post
sds. Thank you for pointing that out.

In my data the pre-treatment SDs are not similar (neither are the
pre-treatment means) between the groups. That is why I am a bit unsure
regarding the most appropriate ES computation method. I have mostly
meta-analysed RCT data in which the two groups are almost identical at
pre-treatment. I could not really find examples in which the difference in
treatment response is being compared between two different populations
receiving the same treatment.

In any case, the following is an okay representation of the actual data and
depending on which ES computation approach I use, the result looks quite
different. So, I was wondering if you could help me in deciding which of
the two is more appropriate or perhaps there is a third option. The data
(or a subsection of it) can also be meta-analysed with raw effect sizes
which does lead to different conclusions as well compared to the SMCRH/SMCC
approach).

Thank you very much in advance,
Mika

### dataset
set.seed(123)
n_G1 <- rpois(50, lambda = 50)
n_G2 <- n_G1

postm_G1 <- rnorm(50, mean = 14, sd = 2.5)
prem_G1 <- rnorm(50, mean = 10, sd = 2)
postsd_G1 <- rnorm(50, mean = 2.4, sd = 0.4)
presd_G1 <- rnorm(50, mean = 1.9, sd = 0.3)

postm_G2 <- rnorm(50, mean = postm_G1 - 7.5, sd = 1.8)
prem_G2 <- rnorm(50, mean = prem_G1 - 5, sd = 1.2)
postsd_G2 <- rnorm(50, mean = 1.2, sd = 0.2)
presd_G2 <- rnorm(50, mean = 1, sd = 0.2)

G <- data.frame(prem_G1,presd_G1, postm_G1, postsd_G1, n_G1,
prem_G2,presd_G2, postm_G2,postsd_G2, n_G2)
G


# Option 1 (could be SMCC as well I suppose)
G1 <- escalc(measure="SMCRH", m1i=postm_G1, m2i=prem_G1, sd1i=presd_G1,
ni=n_G1, sd2i = postsd_G1, ri=c(rep(0.7,50)), data=G)
G2 <- escalc(measure="SMCRH", m1i=postm_G2, m2i=prem_G2, sd1i=presd_G2,
ni=n_G2, sd2i = postsd_G2, ri=c(rep(0.7,50)), data=G)
dat <- data.frame(yi = G1$yi - G2$yi, vi = G1$vi + G2$vi)
dat
# Crude mean of Effect sizes
mean(dat$yi)

# Option 2
pldpre_sd = sqrt((presd_G1^2 + presd_G2^2) / 2)
ES = ((postm_G1 - prem_G1) - (postm_G2 - prem_G2)) / pldpre_sd
ES
# Crude mean of Effect sizes from this formula
mean(ES)


to 16. maalisk. 2023 klo 17.36 Viechtbauer, Wolfgang (NP) (
wolfgang.viechtbauer at maastrichtuniversity.nl) kirjoitti:

Hi Mika,

Depends on what you mean by 'best'. But note that escalc(measure="SMCRH,
...) computes (m1i-m2i)/sd1i, so you are using the post-treatment SD to
standardize, which is a bit unusual and different from your second approach
where you use the average pre-treatment SD to standardize. This aside, the
two approaches should lead to rather similar estimates, especially if the
pre-treatment SDs (assuming those are used in both approaches) are similar
across the two groups.

Best,
Wolfgang

-----Original Message-----
From: R-sig-meta-analysis [mailto:

r-sig-meta-analysis-bounces at r-project.org] On

Behalf Of Mika Manninen via R-sig-meta-analysis
Sent: Monday, 13 March, 2023 17:28
To: R meta
Cc: Mika Manninen
Subject: [R-meta] Computing Effect Size for Difference in Differences with
Different Populations

Dear community,

I am currently working on a meta-analysis that aims to examine the
difference in training effects between two populations. Both
populations underwent the same training, but at pre-test, the groups
have significantly different means and standard deviations (about
1-2sd difference in means).

I am interested in computing the effect size for the difference in
differences between the two groups. Specifically, I would like to know
what is the best way to calculate the effect size given the
significant difference in means and standard deviations at pre-test.

Would the below be roughly accurate (Option 1):

Option 1.

G1 <- escalc(measure="SMCRH", m1i=postm_G1, m2i=prem_G1,
sd1i=postsd_G1,ni=n_G1, sd2i = presd_G1, ri=c(rep(0.7,10)), data=G)
G2 <- escalc(measure="SMCRH", m1i=postm_G2, m2i=prem_G2,
sd1i=postsd_G2, ni=n_G2, sd2i = presd_G2, ri=c(rep(0.7,10)), data=G)
dat <- data.frame(yi = G1$yi - G2$yi, vi = G1$vi + G2$vi)

Option 2.

ES = (G1 post_mean - G2 pre_mean) - (G2 post_mean - G2 pre_mean) /

pldpre_sd

pldpre_sd = sqrt((presdG1^2 + presdG2^2) / 2)

### dataset

set.seed(123)

postm_G1 <- rnorm(100, mean = 14, sd = 2.5)
prem_G1 <- rnorm(100, mean = 10, sd = 2)
postsd_G1 <- rnorm(100, mean = 1.4, sd = 0.2)
presd_G1 <- rnorm(100, mean = 1, sd = 0.2)
n_G1 <- rpois(100, lambda = 50)

postm_G2 <- rnorm(100, mean = 7.5, sd = 1.8)
prem_G2 <- rnorm(100, mean = 5, sd = 1.2)
postsd_G2 <- rnorm(100, mean = 0.9, sd = 0.2)
presd_G2 <- rnorm(100, mean = 0.6, sd = 0.2)
n_G2 <- rpois(100, lambda = 50)

G <- data.frame(postm_G1, prem_G1, postsd_G1, n_G1, presd_G1,
postm_G2, prem_G2, postsd_G2, n_G2, presd_G2)

###

Thank you in advance for your time and help.

Best wishes,
Mika

[R-meta] Computing Effect Size for Difference in Differences with Different Populations

Thread (5 messages)