[R-meta] Correcting Hedges' g vs. Log response ratio in nested studies

Hi James,

If you look at Eq. number E.5.1 on p1 of this document: (
https://ies.ed.gov/ncee/wwc/Docs/referenceresources/WWC-41-Supplement-508_09212020.pdf)
they define the correction factor as: sqrt( 1-((2*(n-1)*icc)/(N-2)) )
where N is n1 + n2 (total sample size), and n as s the average number of
individuals per cluster.

Am I missing something? Or is the correction factor linked above from WWC
inaccurate?

Thank you,
Yuhang

On Thu, Nov 2, 2023 at 1:51?PM James Pustejovsky <jepusto at gmail.com>
wrote:

Responses inline below.

On Thu, Nov 2, 2023 at 3:30?PM Yuhang Hu <yh342 at nau.edu> wrote:

Regarding your first message, it looks like the correction factor for
SMD is: sqrt( 1-((2*(n-1)*icc)/(N-2)) ) where n is the average cluster size
for each comparison in a study, and N is the sum of the two groups' sample
sizes. So, I wonder how the number of clusters is impacting the correction
factor for SMD as you indicated?

N = n * m, where m is the number of clusters. So the correction factor is

sqrt( 1-((2*(n-1)*icc)/(m * n - 2)) ~=  sqrt( 1- 2 * icc /m)

Regarding my initial question, my hunch was that for SMD, the SMD
estimate and its sampling variance are (non-linearly) related to one
another. Therefore, correcting the sampling variance for a design issue
will necessitate correcting the SDM estimate as well.

On the other hand, the LRR estimate and its sampling variance are not as
much related to one another. Therefore, correcting the sampling variance
for a design issue will not necessitate correcting the LRR estimate as well.

No, the issue you've described here is pretty much unrelated to the bias
correction problem.

On Thu, Nov 2, 2023 at 8:41?AM James Pustejovsky <jepusto at gmail.com>
wrote:

One other thought on this question, for the extra-nerdy.

The formulas for the Hedges' g SMD estimator involve what statisticians
would call "second-order" bias corrections, meaning corrections arising
from having a limited sample size. In contrast, the usual estimator of the
LRR is just a "plug-in" estimator that works for large sample sizes but can
have small biases with limited sample sizes. Lajeunesse (2015;
https://doi.org/10.1890/14-2402.1) provides formulas for the
second-order bias correction of the LRR estimator with independent samples.
These bias correction formulas actually *would* need to be different if you
have clustered observations. So, the two effect size metrics are maybe more
similar than it initially seemed:
- Both metrics have plug-in estimators that are not really affected by
the dependence structure of the sample, but whose variance estimators do
need to take into account the dependence structure
- Both metrics have second-order corrected estimators, the exact form
for which does need to take into account the dependence structure.

James

On Thu, Nov 2, 2023 at 8:14?AM James Pustejovsky <jepusto at gmail.com>
wrote:

Wolfgang is correct. The WWC correction factor arises because the
sample variance is not quite unbiased as an estimator for the total
population variance in a design with clusters of dependent observations,
which leads to a small bias in the SMD.

The thing is, though, this correction factor is usually negligible.
Say you?ve got a clustered design with n = 21 kids per cluster and 20
clusters, and an ICC of 0.2. Then the correction factor is going to be
about 0.99 and so will make very little difference for the effect size
estimate. It only starts to matter if you?re looking at studies with very
few clusters and non-trivial ICCs.

James

On Nov 2, 2023, at 3:04 AM, Viechtbauer, Wolfgang (NP) via

R-sig-meta-analysis <r-sig-meta-analysis at r-project.org> wrote:

?Dear Yuhang,

I haven't looked deeply into this, but an immediate thought I have

is that for SMDs, you divide by some measure of variability within the
groups. If that measure of variability is affected by your study design,
then this will also affect the SMD value. On the other hand, this doesn't
have any impact on LRRs since they are only the (log-transformed) ratio of
the means.

Best,
Wolfgang

-----Original Message-----
From: R-sig-meta-analysis <

r-sig-meta-analysis-bounces at r-project.org> On Behalf

Of Yuhang Hu via R-sig-meta-analysis
Sent: Thursday, November 2, 2023 05:42
To: R meta <r-sig-meta-analysis at r-project.org>
Cc: Yuhang Hu <yh342 at nau.edu>
Subject: [R-meta] Correcting Hedges' g vs. Log response ratio in

nested studies

Hello All,

I know that when correcting Hedges' g (i.e., bias-corrected SMD,

aka "g")

in nested studies, we have to **BOTH** adjust our initial "g" and

its

sampling variance "vi_g"
(

https://ies.ed.gov/ncee/wwc/Docs/referenceresources/WWC-41-Supplement-

508_09212020.pdf).

But when correcting Log Response Ratios (LRR) in nested studies, we

have to

**ONLY** adjust its initial sampling variance "vi_LRR" but not

"LRR" itself

(

https://stat.ethz.ch/pipermail/r-sig-meta-analysis/2021-October/003486.html
).

I wonder why the two methods of correction differ for Hedge's g and

LRR?

Thanks,
Yuhang