[R-meta] Question about Cohen's d and meta-regression

Dear Angeline,

in addition to Wolfgang's excellent advice, here are two strategies that I
have employed successfully:

1) When interpreting d (or Hedges' g), try to convert it back to one of
the original metrics. I.e. if your original scale has a population standard
deviation of s in a typical sample, then  d*s is the shift in points on the
original scale. Depending on context, people find this useful frequently,
especially, when there is a dominant measure in the field.

2) I have used the strategy of Pigott & Hedges for power analysis in the
planning stage of meta-regressions. The book by Pigott is more accessible
than the paper and contains R code which is relatively easy to adapt.
Results will depend strongly on your assumptions, so I would caution you to
be rather conservative and assume smaller studies and larger random
effects. That said, 6 studies might or might not be sufficient.

Best wishes,
   Philipp

On Thu, Jan 4, 2018 at 10:24 AM, Viechtbauer Wolfgang (SP) <
wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:

Dear Angeline,

With respect to 1:

This isn't part of your question, so feel free to ignore this, but I
would be curious to see the details of how you are converting a
within-subjects d to a between-subjects d (including the computation of the
sampling variance).

As for your actual question: In my opinion, neither U3, CLES, nor any
other unitless measure will help with the interpretation of the actual
magnitude of the effect. In his famous 1994 paper ("The earth is round (p <
.05)"), Cohen himself also recommended moving away from 'standardized'
measures of effect and instead advocated working with raw measures of
effect. When meta-analyzing studies that used different measures to begin
with, doing the meta-analysis with raw effects isn't possible, but we can
still try to express the final result in terms of a raw effect. In
particular, take the estimated (average) d value from the meta-analysis and
multiply this with the SD of a scale that is well known/established in the
area where the studies are coming from. For example, suppose a
meta-analysis finds that children exposed to low levels of lead score lower
on cognitive ability tests than non-exposed children with a d of -0.25.
Then using a normed IQ test with an SD of 15, this implies a di
 fference of -0.25 * 15 = -3.75 IQ points. I find this much easier to
interpret than, for example: "The chances that a randomly drawn exposed
child will score lower on a cognitive ablity test than a randomly drawn
non-exposed child is 57%" (this would be the corresponding CLES result).
The only difficulty with this approach might be that there isn't a normed
measure with an 'established' SD. But one can often still identify a
measure that is commonly used in a particular area, look at a bunch of
studies to see what kind of SDs are typically observed, and then pick a
sensible value.

2) Again, this is just my opinion, but such rules of thumb are usually
useless. If I remember correctly, I've also read somewhere that one should
have 5 (or 10) studies per moderator. I suspect these rules have arisen
because analogous rules have been floating around for standard regression
analyses (where they've also been criticized; for example: Maxwell, S. E.
(2000). Sample size and multiple regression analysis. Psychological
Methods, 5(4), 434-458.).

Even with a relatively low value of k/p (k = number of studies, p =
number of model coefficients), the Type I error rate for testing moderators
is close to nominal as long as one employs something like the Knapp &
Hartung correction. The problem is that power is likely to be very low then
as well. One could try to do a proper power analysis (Hedges, L. V., &
Pigott, T. D. (2004). The power of statistical tests for moderators in
meta-analysis. Psychological Methods, 9(4), 426-445.), but I've never seen
this done in practice (at least not reported). Something like a 'k per p'
rule is of course much easier to implement.

Best,
Wolfgang

-----Original Message-----
From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bo
unces at r-project.org] On Behalf Of Angeline Tsui
Sent: Wednesday, 03 January, 2018 20:07
To: r-sig-meta-analysis at r-project.org
Subject: [R-meta] Question about Cohen's d and meta-regression

Hello all,

I have some questions about effect sizes and meta-regression.

1) I am using between-subject Cohen's d (but this effect size is converted
from a within-subject d based on Morris and Deshan (2002) for my
meta-analysis. The main reason of the conversion is to make this mean
effect size comparable to other meta-analysis in my field, especially when
some other meta-analysis may also report between-subject Cohen's d.
However, I have concerns of interpreting the magnitude of the mean effect
size.

When I interpret the mean Cohen's d, I use the Cohen (1988) rule of thumb:
0.2 -> small, 0.5 -> medium and 0.8 -> large. However, a concern of
applying this Cohen's values rule of thumb to the current meta-analysis is
that it is not "context-specific". For example, a small effect size of 0.3
can be regarded as large effect size in some other contexts.

In this case, what other metrics would you recommend me to use to report
the mean effect size magnitude? I have searched some resources online and
see recommendation about the use of Cohen's U3 index or Common language
effect size statistics (CLES)? From what I understand, both U3 index is a
percentile index whereas CLES is a probability index. But I am not
familiar
with these two metrics, can someone give me recommendation which index is
better?

Finally, if I change my effect size to hedge's g because the sample size
of
each study is small and I need to correct it by converting Cohen's d to
hedge's g. Can I still convert the mean hedge's g using the same formula
of
Cohen's U3 or CLES for interpreting the percentile/probability of the mean
effect size?

2) I ran a meta-regression model with which include a number of
moderators.
My concern is the minimum sample size needed for each moderator level. I
was not able to find resources for recommendation of sample size for
moderator analysis. I did learn something from my teacher, she told me
that
the rule of thumb is at least 6 studies for a moderator analysis level
(e.g., Gender is a moderator. Then I need to have at least 6 studies that
examined male and at least 6 studies that examined female). Is this true?
I
again was not able to find this rule of thumb from books.

Thanks,
Angeline

--
Prof. Dr. Philipp Doebler
Technische Universit?t Dortmund
Fakult?t Statistik
Vogelpothsweg 87
<https://maps.google.com/?q=Vogelpothsweg+87+44227+Dortmund&entry=gmail&source=g>
44227 Dortmund
<https://maps.google.com/?q=Vogelpothsweg+87+44227+Dortmund&entry=gmail&source=g>

Tel.: +49 231-755 8259 <+49%20231%207558259>
Fax: +49 231-755 3918 <+49%20231%207553918>
doebler at statistik.tu-dortmund.de
www.statistik.tu-dortmund.de/1261.html


Wichtiger Hinweis: Die Information in dieser E-Mail ist vertraulich. Sie
ist ausschlie?lich f?r den Adressaten bestimmt. Sollten Sie nicht der
f?r diese E-Mail bestimmte Adressat sein, unterrichten Sie bitte den
Absender und vernichten Sie diese Mail. Vielen Dank.
Unbeschadet der Korrespondenz per E-Mail, sind unsere Erkl?rungen
ausschlie?lich final rechtsverbindlich, wenn sie in herk?mmlicher
Schriftform (mit eigenh?ndiger Unterschrift) oder durch ?bermittlung
eines solchen Schriftst?cks per Telefax erfolgen.

Important note: The information included in this e-mail is confidential.
It is solely intended for the recipient. If you are not the intended
recipient of this e-mail please contact the sender and delete this
message. Thank you.
Without prejudice of e-mail correspondence, our statements are only
legally binding when they are made in the conventional written form
(with personal signature) or when such documents are sent by fax.