[R-meta] SMD from three-level nested design (raw data available)

Fabian,

The overarching goal in this context is to choose an effect size parameter
that is as comparable as possible to the other studies in the synthesis.
Three scenarios:

1. If those other studies are mostly individually randomized experiments
conducted across multiple contexts, but without the repeated measures
component, then I would argue that d_T (the average effect, standardized
based on the total variance of the outcome) might be more appropriate. The
reason is that the distribution of observed outcomes will be comprised of
both between-person _and within-person (between-trial)_ variation. If
participants respond to an instrument only once, then there is still some
unreliability in the resulting scores, so the corresponding variance
component should be included in the denominator.

2. If the other studies are mostly individually randomized experiments
conducted across narrow contexts, then it might make sense to use d_WS (eq.
18.35 in Hedges, 2009), which excludes the between-group variation from the
denominator of the effect size. The reasoning here is that if the other
studies use samples that would end up as a single group in the
cluster-randomized trial, then the distribution of observed outcomes in
those studies will not include the between-group variation. For instance,
say that study A randomized at the school level, whereas studies B, C,
D,... used samples from a single school each. Then the latter studies won't
have between-school variation in the outcome, and we would exclude the
between-school component from study A in order to maintain comparability
with the other studies.

3. If the other studies mostly DID use repeated measures, but averaged the
scores together before analysis, then the distribution of observed outcomes
in those studies will not include the within-participant variation (or
actually it will but to a much-reduced extent). In this situation, it would
make sense to exclude the within-participant variance component from the
denominator of the effect size (and thus include only the
between-participant or the sum of the between-participant and between-group
variance components, depending on considerations analogous to the above).
But note that Hedges (2009) sees these effect sizes as less likely to be of
general interest (see notes on p. 348).

James

On Mon, Nov 5, 2018 at 5:31 PM Fabian Schellhaas <fabian.schellhaas at yale.edu>
wrote:

[R-meta] SMD from three-level nested design (raw data available)

Thread (7 messages)