[R-meta] Combining studies with different within-subject designs

Fri, Feb 18, 2022 2:10 AM

Dear Pablo,

I am confused why you call case 2 a 'within-study design'. That looks like a two-group post-test only design to me.

In any case, for all three cases, you want to use 'raw score standardization' to make the various effect size measures comparable, at least under some restrictive assumptions, such as that there is no inherent time effect or time by group interaction.

So, for case 2, you compute standard SMDs, which use the SD at a single timepoint (i.e. at post). This is in essence 'raw score standardization'.

Ananlogously, for case 3, you also use 'raw score standardization', so measure "SMCR" in escalc() lingo.

And finally, for case 1, you again want to standardize based on the SD of a single timepoint, not the SD of the change scores. See:

https://www.metafor-project.org/doku.php/analyses:morris2008

for a discussion of how to do this.

If authors do not report the required raw score SDs, then you will have to get creative getting these (e.g., contacting authors, back-calculating them based on the SD of the change scores and other information provided, guestimating them).

I would also code the type of design that was used in each study (and hence the type of effect size measure that was computed) to examine whether there are still systematic differences between these groups, even if the type of standardization was the same across measures.

Best,
Wolfgang

-----Original Message-----
From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces at r-project.org] On
Behalf Of Pablo Grassi
Sent: Wednesday, 16 February, 2022 16:14
To: r-sig-meta-analysis at r-project.org
Subject: [R-meta] Combining studies with different within-subject designs

Dear all,

Maybe some of you can help me out with the following design-conundrum. I
am currently performing a series of meta-analysis investigating the
effect on an intervention in closely related outcomes. Most of the
reviewed studies are within-subject designs (one group of participants),
for which the standardized mean differences (SMD; Hedge's g) are
differences of change scores divided by the SD of the difference (with
change score standardization), as follows:

??? SMD = M_diff / SDdiff?? (for simplicity in this E-mail without the
bias-correction factor)

Unfortunately, there is a huge variability in the control measurements
used. Roughtly following the nomenclature from Morris 2008, I have the
following different design-cases:

??? Case 1) Within-subject desing, pre-post-control (WS_PPC):

?????????????? M_diff_ws_pcc = (post_Treatment - pre_Treatment) -
(post_Control - pre_Control)

However, some others had no baseline pre-intervention measurement (i.e.
only report post-intervention measurements), i.e. Post-test only with
control design (WS_POWC)

??? Case 2) M_diff_ws_powc = (post_Treatment - post_Control)

And some few others just have a post-pre measurement but no control
(i.e. only report a change score, single-group pre-post, SGPP), so that:

??? Case 3) M_diff_ws_sgpp = (post_Treatment - pre_Treatment)

Then, while the M_diff if Case 1 and 2 is measuring +/- the same effect,
their SDdiffs (and thus SMDs) are not really comparable as they reflect
different things.

Thus:

 * What would be the "standard" approach to include the within-subject
   design studies (Case 1, 2, 3) in the same meta-analysis, if this is
   possible at all? (Please consider that most of the publications
   report ONLY the SD of the change scores and NOT the SD of the pre
   and post conditions separately or in case 2 of the post-interventions).

Best,

Pablo

[R-meta] Combining studies with different within-subject designs

Thread (4 messages)