Mikkel,
Thank you. Is this new formula (with F-value) for the sampling variance of
g in the WWC procedures handbook? Would you please provide a reference?
Best regards,
Tim M
On Fri, Oct 1, 2021 at 3:22 PM Mikkel Vembye <mikkel.vembye at gmail.com>
wrote:
When I have relevant F-values from ANCOVA and related models, I calculate
vg_corrected = omega^2 * [(g_corrected^2/F-value) * eta +
g_corrected^2/(2*h)]
Have a nice weekend.
Best,
Mikkel
Den fre. 1. okt. 2021 kl. 21.41 skrev Mikkel Vembye <
mikkel.vembye at gmail.com>:
Sorry. I referred to an older version of the Appendix. I usually just
follow WWC's recommendation when I cannot obtain R2. This is
"if R2 is not available, then WWC will take a cautious approach to
calculating the
standard error and assume a value of zero for R2. This cautious
approach will overestimate the
magnitude of the standard error but protects against type I error." (See
the WWC Procedure Handbook, p. E-5)
https://ies.ed.gov/ncee/wwc/Docs/referenceresources/WWC-Procedures-Handbook-v4-1-508.pdf
Whether this approach is better, is more a James question.
Mikkel
Den fre. 1. okt. 2021 kl. 19.44 skrev Timothy MacKenzie <
fswfswt at gmail.com>:
Much appreciated, Mikkel. I saw that. BTW, there is no Table 5, it's a
typo in the WWC document (I found other typos as well).
But I have both ANOVAs and a few ANCOVAs from primary studies that did
cluster assignment but ignored nesting structure, with barely any R^2
reported in them.
My understanding is that I should find a more general SE[g] that only
requires icc, am I correct in thinking this way?
Thanks,
Tim M
On Fri, Oct 1, 2021 at 12:32 PM Mikkel Vembye <mikkel.vembye at gmail.com>
wrote:
Hi Tim,
Glad that I/we can help. You find the ANCOVA examples (both
uncorrected and corrected) in Table 3.
[image: image.png]
I forgot to mention that you also can find some corrections to Hedges
(2007) in Table 5.
All the best,
Mikkel
Den fre. 1. okt. 2021 kl. 19.21 skrev Timothy MacKenzie <
fswfswt at gmail.com>:
Dear James and Mikkel,
Thank you both. In my case, the primary studies have used AN(C)OVAs
with (non-)random assignment of classes to conditions. I have the
Means and SDs of student-level data for conditions.
Based on:
https://ies.ed.gov/ncee/wwc/Docs/referenceresources/WWC-41-Supplement-508_09212020.pdf
, I should use the unbiased version of equation E.5.1 to compute the
SMD effect size (g):
g = wb / s * sqrt( 1 - (2*(n-1)*icc / N - 2) ) ; s = pooled standard
deviation; n = ave. cluster size; N = n_t + n_c; w = hedges'
correction factor
Based on the same document, the standard error of g (SE[g]) for
cluster-assignment studies is (equation E.7.1 under "Cluster
assignment"):
SE[g] = w * sqrt( (SE_uc / s )^2 * eta + (g^2 / (2*h)) ); eta = 1 +
(n - 1)*icc; h = ( [(N-2)-2*(n-1)*icc ]^2 ) / ((N-2)*(1-icc)^2 +
2*(N-2*n)*icc*(1-icc) )
where SE_uc = regression coefficient standard errors uncorrected for
clustering in the primary studies.
Am I pointing to the correct formulas? If yes, I don't have SE_uc in
my primary studies, what should I do?
Thanks,
Tim M
------ Forwarded Message ------
From: Mikkel Vembye <mikkel.vembye at gmail.com>
Date: Fri, Oct 1, 2021 at 6:54 AM
Subject: Re-re: [R-meta] Meta-analyzing studies that failed to account
for their nested data
To: <fswfswt at gmail.com>, <r-sig-meta-analysis at r-project.org>
Hi Tim,
Just to follow up on James, WWC do also have a nice description of how
they handle cluster trials and quasi-experiments:
https://ies.ed.gov/ncee/wwc/Docs/referenceresources/WWC-41-Supplement-508_09212020.pdf
Mikkel
On Thu, Sep 30, 2021 at 11:09 PM James Pustejovsky <jepusto at gmail.com>
wrote:
For studies that claim to find negligible ICCs, I would guess that
they base this judgement either on a) failing to reject a test of ICC = 0
or b) a rule of thumb. a) is not a good justification because with few
classes, the test will have little power. b) is arbitrary and even small
ICCs (of say 0.02 or 0.04) can be consequential for estimating the variance
of the effect size estimate. I would use the ICC adjustment regardless.
To your second question, yes these adjustments are also important
James
On Thu, Sep 30, 2021 at 10:53 PM Timothy MacKenzie <
fswfswt at gmail.com> wrote:
Dear James,
Many thanks for this information. Certainly this is serious.
I should add that a few of the (newer) studies in my pool say that
they found their ICCs to be negligible and opted for the
analyses (maybe I should not adjust the sampling variances in these
cases, correct?).
Also, I'm assuming that I can use these sampling variance
for quasi-experiments where schools/centers themselves haven't been
randomly recruited as well?
Thanks,
Tim M
On Thu, Sep 30, 2021 at 9:40 PM James Pustejovsky <
jepusto at gmail.com> wrote:
Hi Tim,
One important issue here is that the sampling variance of the
effect size estimate calculated from such a study will be
inaccurate---possibly even an order of magnitude smaller than it should be.
If you ignore this, the consequence will be to make the effect size
estimates appear far more precise than they actually are.
To properly correct the sampling variance estimate, you would
need to know the intra-class correlation describing the proportion of the
total variation in the outcome that is at the cluster level (in this case,
what fraction of the total variance is between classes?). If this isn't
reported, then it may be possible to develop a reasonable estimate based on
external information. The Cochrane Handbook describes how to correct the
sampling variance based on an imputed intra-class correlation:
James
On Thu, Sep 30, 2021 at 4:58 PM Timothy MacKenzie <
fswfswt at gmail.com> wrote:
Hello All,
I've noticed almost all the studies I have selected for
have ignored the nested structure of their data (subjects
classrooms) and have conducted only single-level analyses.
I've extracted the condition-level summaries from those studies
Means and SDs for C vs. T groups).
But I'm wondering if I can/should make any adjustment to my
meta-regression model to account for the nested structure of
in those studies AND if not, whether such a situation poses a
limitation to my meta-analysis?
Thank you very much for your assistance,
Tim M