[R-meta] SMD from three-level nested design (raw data available)

I don't know of any further work on this beyond the Hedges chapter and the
two papers on which it is based. Anyone else have pointers?

On Tue, Nov 6, 2018 at 10:49 AM Fabian Schellhaas <
fabian.schellhaas at yale.edu> wrote:

Dear James,

Thanks for these clarifications, this helps a lot. Scenario 1 does indeed
apply here, so I will add the within-subjects variance component to the
denominator. Is there any further reading you would recommend on this topic?

Many thanks,
Fabian

---
Fabian M. H. Schellhaas | Ph.D. Candidate | Department of Psychology |
Yale University


On Tue, Nov 6, 2018 at 11:30 AM James Pustejovsky <jepusto at gmail.com>
wrote:

Fabian,

The overarching goal in this context is to choose an effect size
parameter that is as comparable as possible to the other studies in the
synthesis. Three scenarios:

1. If those other studies are mostly individually randomized experiments
conducted across multiple contexts, but without the repeated measures
component, then I would argue that d_T (the average effect, standardized
based on the total variance of the outcome) might be more appropriate. The
reason is that the distribution of observed outcomes will be comprised of
both between-person _and within-person (between-trial)_ variation. If
participants respond to an instrument only once, then there is still some
unreliability in the resulting scores, so the corresponding variance
component should be included in the denominator.

2. If the other studies are mostly individually randomized experiments
conducted across narrow contexts, then it might make sense to use d_WS (eq.
18.35 in Hedges, 2009), which excludes the between-group variation from the
denominator of the effect size. The reasoning here is that if the other
studies use samples that would end up as a single group in the
cluster-randomized trial, then the distribution of observed outcomes in
those studies will not include the between-group variation. For instance,
say that study A randomized at the school level, whereas studies B, C,
D,... used samples from a single school each. Then the latter studies won't
have between-school variation in the outcome, and we would exclude the
between-school component from study A in order to maintain comparability
with the other studies.

3. If the other studies mostly DID use repeated measures, but averaged
the scores together before analysis, then the distribution of observed
outcomes in those studies will not include the within-participant variation
(or actually it will but to a much-reduced extent). In this situation, it
would make sense to exclude the within-participant variance component from
the denominator of the effect size (and thus include only the
between-participant or the sum of the between-participant and between-group
variance components, depending on considerations analogous to the above).
But note that Hedges (2009) sees these effect sizes as less likely to be of
general interest (see notes on p. 348).

James

On Mon, Nov 5, 2018 at 5:31 PM Fabian Schellhaas <
fabian.schellhaas at yale.edu> wrote:

Dear James,

Thanks so much for your reply, this is really helpful and made me think
carefully about the data I'm dealing with. The effect I'm trying to compute
is defined by Hedges (2009, p. 348) as d_BC, i.e. the treatment effect at
level 2 of a 3-level design. In "my" dataset, the unit of measurement is
the allocation decision (level 1), and the unit of randomization is the
group (level 3). The effect I'm after, however, is the treatment effect at
the level of the participant (level 2).

Unfortunately, Hedges (2009) does not provide the equation for the
computation of d_BC using fixed-effect estimates and variance components.
However, in the context of a 2-level model, Hedges (2009) defines the
between-cluster effect as

d_B = b / sig_B  [Eq. 18.17]

where b is the estimated fixed effect and sig_B^2 is the
between-cluster variance component. Note that the within-cluster variance
component is omitted from the denominator. By contrast, the total treatment
effect is defined as

d_T = b / sqrt(sig_B^2 + sig_W^2)  [Eq. 18.23]

where b is again the estimated fixed effect, sig_B^2 is the
between-cluster variance component, and sig_W^2 is the within-cluster
variance component. I tried to apply this logic to the study I'm coding, in
which the effect size of interest is not the total treatment effect, but
rather the treatment effect at the level of individual participants (level
2). As such, I omitted sig_w from the denominator. My understanding is that
if I add the repeated-measures variance component to the denominator, as
you suggested, I would get the treatment effect at the level of the
allocation decision (as per Hedges, 2009, Eq. 18.55). And wouldn't such an
effect size be incomparable to the other SMDs in the meta-analysis, which
represent a treatment effect at the level of participants?

Many thanks for your help,
Fabian

---
Reference:
Hedges, L. V. (2009). Effect sizes in nested designs. In Cooper, H.,
Hedges, L. V., & Valentine, J. C. (Eds.), The Handbook of Research
Synthesis and Meta-Analysis (pp. 337-355). New York: Russell Sage
Foundation.


On Sun, Nov 4, 2018 at 10:49 PM James Pustejovsky <jepusto at gmail.com>
wrote:

Fabian,

Your calculations make sense to me for a two-level model (participants
nested within groups), but you've described a three-level model. What
happened to the other level (repeated measures, nested within
participants)? If you have a positive variance component estimate for it,
then I think it would make sense to include it in the denominator of the
effect size. If X is the estimated variance of the repeated measures nested
within participant, then take

d = 6.95 / sqrt(X + 143.64 + 217.17)

James

On Sat, Nov 3, 2018 at 3:22 PM Fabian Schellhaas <
fabian.schellhaas at yale.edu> wrote:

Hi all,

I have a question about computing a standardized mean difference
(SMD) from
a primary study with a three-level nested design. The study in
question
randomly assigned groups of participants to a treatment or control
condition, and then measured individual participants' resource
allocations.
While some respondents made only one such decision, others made two.
As
such, the data in this study has three levels: resource allocation
decisions, which are nested in participants, which in turn are nested
in
groups.

I would like to compute an effect size that reflects the
between-participant effect of treatment vs. control. I have the raw
data,
which the authors luckily made available. As such, I can easily fit a
linear mixed model with a fixed effect for treatment vs. control, and
a
nested random effect to account for the three-level design. However,
how do
I extract a SMD from the fitted model that is comparable to SMDs from
single-level designs?

The estimate for the fixed effect is 6.95, with a SE of 6.27. The
variance
components of the random effects are 143.64 for participant nested in
group, and 217.17 for group. Based on formula 18.17 in Hedges (2009),
I
believe I would compute *d* = 6.95/sqrt(143.64 + 217.17) = 0.366.
However,
I would like to confirm that this is indeed the correct approach
before I
proceed.

Many thanks!
Fabian

---
Fabian M. H. Schellhaas | Ph.D. Candidate | Department of Psychology
| Yale
University

        [[alternative HTML version deleted]]