-----Original Message-----
From: R-sig-meta-analysis <r-sig-meta-analysis-bounces at r-project.org> On Behalf
Of Zhouhan Jin via R-sig-meta-analysis
Sent: Thursday, May 9, 2024 05:24
To: James Pustejovsky <jepusto at gmail.com>
Cc: Zhouhan Jin <zjin65 at uwo.ca>; R Special Interest Group for Meta-Analysis <r-
sig-meta-analysis at r-project.org>
Subject: Re: [R-meta] Correcting gain effects in nested studies
Hi James,
Why should ICC be smaller for change scores, because subtraction removes some of
the dependency with clusters, kind of like centering within clusters?
Anyway, knowing now that nothing exists for SMCC, I am actually open to
standardizing my pre-post mean changes using other denominators including
standard deviation of the pre-test as in SMCR (sidenote: why escalc() then needs
"ri=" to calculate an SMCR estimate?).
My data only consists of pre-post Means and their respective Sds.
But, what equation in the WWC handbook specifically gives the formula to adjust
the SMCRs and its SEs calculated by metafor::escalc() for (one level of)
nesting?
WWC's notations are often confusing.
Best wishes,
Zhouhan
On May 8, 2024 at 21:59 -0400, James Pustejovsky <jepusto at gmail.com>, wrote:
Ah, sorry I missed that you want SMCC rather than SMCR. Taylor and WWC focus on
SMCR. I'm not aware of anything developed specifically for SMCC. Apart from the
regular critiques of SMCC, there's also the difficulty of determining an
appropriate ICC for the change scores. Just speculating, but I would guess that
ICCs for change scores might be quite different---and probably smaller?---than
ICCs for outcomes measured cross-sectionally (for which there are many sources
of empirical data available).
James
On Wed, May 8, 2024 at 8:58?AM Zhouhan Jin <zjin65 at uwo.ca<mailto:zjin65 at uwo.ca>>
wrote:
Thank you, James! I might be missing something, but I couldn't specifically
locate the formulas to adjust SMCC (and its SE) in ?metafor::escalc, certainly
in Taylor et al.
Regarding WWC, is there a specific equation that allows for nestedness-adjusting
of SMCC (defined below)?
SMCC = (postM - preM) / SD_of_change_scores
with its SE in metafor by default (vtype='LS') being calculated without a small-
sample correction as:
SE_SMCC = sqrt( (1/n) + ((SMCC^2) / (2*n)) )
Many thanks!
Best wishes,
Zhouhan
On May 7, 2024 at 21:30 -0400, James Pustejovsky
<jepusto at gmail.com<mailto:jepusto at gmail.com>>, wrote:
See Taylor, Pigott, and Williams (2022;
https://doi.org/10.3102/0013189X211051319) for how to handle cluster-randomized
trials that involve gain scores or covariate adjustment. They provide a shiny
app too. The technical details are also described in Appendix E of the What
Works Clearinghouse handbook (Version 5;
https://ies.ed.gov/ncee/WWC/Docs/referenceresources/Final_WWC-HandbookVer5_0-0-
508.pdf). See pp. 173-174
The methods described in these sources are consistent with the "general recipe"
for standardized mean difference estimates as described here:
https://www.jepusto.com/alternative-formulas-for-the-smd/
James
On Tue, May 7, 2024 at 7:14?PM Zhouhan Jin <zjin65 at uwo.ca<mailto:zjin65 at uwo.ca>>
wrote:
Hello All,
Hedges (2007) provides formulas for adjusting SMD effects (g) and their SEs for
when primary studies have a nested design (below).
But I want to compute gain effects (ex. SMCC in metafor::escalc) from my nested
studies, not SMDs.
So, how can I adjust my SMCCs and their SEs for nestedness in the primary
studies?
adjusted_g = g * sqrt(1 - ((2 * (n_bar - 1) * icc) /
(n_cluster * n_bar - 2)))
adjusted_SE = ((Nt+Nc)/(Nt*Nc))*(1 + ((n_bar- 1)*icc)) +
( g^2 * (
(((N_tot -2)*(1-icc)^2 ) + (n_bar*(N_tot - 2*n_bar)*icc^2) +
(2* (N_tot - 2*n_bar) * icc * (1 - icc)) ) /
((2* (N_tot-2)) * ( (N_tot-2) - (2* (n_bar-1)*icc) ))
) )
Thanks a lot!
Best wishes,
Zhouhan