[R-meta] SMD Metric
Thank you, James.
I think we usually treat items on a test/instrument to be equivalent (all
other things equal), if they differ from one another by some additive and
multiplicative constants. We can probably extend this inter-item
equivalency concept to equate different tests (all other things equal).
It seems some effect sizes (e.g., SMD and correlation coef.) take this into
account and 'work best' if such equivalence holds and work less well as
this equivalence weakens.
But other effect sizes seem to be somewhat insensitive to such equivalency.
ROM seems to be one of them. But do you think that it is desirable for ROM
to not give the same estimates for when such an equivalence, in fact,
exists:
x1 <- rnorm(50, 36, 6)
x2 <- rnorm(50, 33, 6)
library(metafor)
escalc(measure="ROM", m1i=mean(x1), sd1i=sd(x1), n1i=length(x1),
m2i=mean(x2), sd2i=sd(x2), n2i=length(x2))
x1 <- 40 + x1 * 3
x2 <- 40 + x2 * 7
escalc(measure="ROM", m1i=mean(x1), sd1i=sd(x1), n1i=length(x1),
m2i=mean(x2), sd2i=sd(x2), n2i=length(x2))
Many thanks,
Yuhang
PS. With a lack of linear equatability, ROM may probably be preferred over
SMD. However, I wonder what to replace COR with, when there is a lack of
linear equatability? I would think in that case there is no alternative to
COR and we need to resort to adding methodological moderators.
On Sun, Apr 2, 2023 at 3:18?PM James Pustejovsky <jepusto at gmail.com> wrote:
Hi Yuhang, On the relationship between linear equatability and SMDs, this article has a good discussion (and also see reference therein): Hedges, L. V. (2008). What are effect sizes and why do we need them?. *Child development perspectives*, *2*(3), 167-171. https://doi.org/10.1111/j.1750-8606.2008.00060.x Just to clarify, my post is NOT implying that the linear equatability assumption is a requirement for using standardized mean differences. I said only that it is "an ideal case" if linear equatability holds. I agree with Wolfgang that, in practice, linear equatability is unlikely to hold in the strict sense. The possible reasons that you listed are all right on. James On Sun, Apr 2, 2023 at 4:00?PM Yuhang Hu via R-sig-meta-analysis < r-sig-meta-analysis at r-project.org> wrote:
Dear Wolfgang, Thank you so much for your response. I would imagine that linear equatability is likely required for the use of many other effect sizes (e.g., correlation coefficients), right? But is/are there possibly some reference(s) discussing the 'linear equatability requirement' for the use of SMD or any other effect sizes (or perhaps any additional considerations like the ones I mention below)? You noted that "But many scales/instruments/questionnaires do not exhibit such strict linear equatability". I wonder what are the underlying reasons for that? For instance, the lack of linear equatability is because the instrument across studies (A) could target slightly different constructs (so their latent constructs differs in location and scale by a bit), or (B) they differ in length or time allowed to respond to the items (and thus in reliability), or (C) the items across the instruments differ in degrees of item difficulty and discrimination, or perhaps (D) the items across the instruments differ in their scale of measurement (one binary, another Likert scale etc.) and thus respondents' responses to the items across the instruments are distributed differently (one binomially distributed, another ordered-categorically distributed etc.) Thank you again, for your help, Yuhang On Sun, Apr 2, 2023 at 10:56?AM Viechtbauer, Wolfgang (NP) < wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:
Dear Yuhang, Essentially, it means that the values on one instrument are assumed to
be
a linear transformation of the values on another instrument. For
example,
say we have measured two groups using scale/instrument/questionnaire A
and
we find:
x1 <- rnorm(50, 36, 6)
x2 <- rnorm(50, 33, 6)
library(metafor)
escalc(measure="SMD", m1i=mean(x1), sd1i=sd(x1), n1i=length(x1),
m2i=mean(x2), sd2i=sd(x2), n2i=length(x2))
Now imagine that instead of A, we had used another
scale/instrument/questionnaire B and that the values on that instrument
are
simply a linear transformation of the scores that would have been
obtained
on A:
x1 <- 40 + x1 * 3
x2 <- 40 + x2 * 3
escalc(measure="SMD", m1i=mean(x1), sd1i=sd(x1), n1i=length(x1),
m2i=mean(x2), sd2i=sd(x2), n2i=length(x2))
As you can see, the SMD values are identical then.
So if values on different instruments are linearly equatable, then it
doesn't matter if we use A or B, the 'effect size' would be identical.
But many scales/instruments/questionnaires do not exhibit such strict
linear equatability. In that case, SMD values may be systematically
higher/lower depending on the instrument used and we end up with a
measurement artifact in our meta-analysis.
I hope that this clarifies things.
Best,
Wolfgang
-----Original Message----- From: R-sig-meta-analysis [mailto:
r-sig-meta-analysis-bounces at r-project.org] On
Behalf Of Yuhang Hu via R-sig-meta-analysis Sent: Sunday, 02 April, 2023 19:21 To: R meta Cc: Yuhang Hu Subject: [R-meta] SMD Metric Hi Everyone, I had a question about the SMD effect size. I read on James' website
that:
"The ideal case for using the SMD metric is when the outcomes in
different
studies are linearly equatable. However, if outcomes exhibit
mean-variance
relationships, linearly equatability seems rather implausible." I was wondering what is meant by linear equatability in the outcomes in different studies and why is that needed for the use of SMD? How could
the
outcomes in different studies be perhaps non-linearly equatable or not equatable at all (neither linearly nor non-linearly)? (I also appreciate reference(s) that discuss such a requirement for the
use
of the SMD metric) Thank you very much for your assistance, Yuhang
[[alternative HTML version deleted]]
_______________________________________________ R-sig-meta-analysis mailing list @ R-sig-meta-analysis at r-project.org To manage your subscription to this mailing list, go to: https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis