[R-meta] Question about Cohen's d and meta-regression

Dear Angeline,

With respect to 1:

This isn't part of your question, so feel free to ignore this, but I would
be curious to see the details of how you are converting a within-subjects d
to a between-subjects d (including the computation of the sampling
variance).

As for your actual question: In my opinion, neither U3, CLES, nor any
other unitless measure will help with the interpretation of the actual
magnitude of the effect. In his famous 1994 paper ("The earth is round (p <
.05)"), Cohen himself also recommended moving away from 'standardized'
measures of effect and instead advocated working with raw measures of
effect. When meta-analyzing studies that used different measures to begin
with, doing the meta-analysis with raw effects isn't possible, but we can
still try to express the final result in terms of a raw effect. In
particular, take the estimated (average) d value from the meta-analysis and
multiply this with the SD of a scale that is well known/established in the
area where the studies are coming from. For example, suppose a
meta-analysis finds that children exposed to low levels of lead score lower
on cognitive ability tests than non-exposed children with a d of -0.25.
Then using a normed IQ test with an SD of 15, this implies a di
 fference of -0.25 * 15 = -3.75 IQ points. I find this much easier to
interpret than, for example: "The chances that a randomly drawn exposed
child will score lower on a cognitive ablity test than a randomly drawn
non-exposed child is 57%" (this would be the corresponding CLES result).
The only difficulty with this approach might be that there isn't a normed
measure with an 'established' SD. But one can often still identify a
measure that is commonly used in a particular area, look at a bunch of
studies to see what kind of SDs are typically observed, and then pick a
sensible value.

2) Again, this is just my opinion, but such rules of thumb are usually
useless. If I remember correctly, I've also read somewhere that one should
have 5 (or 10) studies per moderator. I suspect these rules have arisen
because analogous rules have been floating around for standard regression
analyses (where they've also been criticized; for example: Maxwell, S. E.
(2000). Sample size and multiple regression analysis. Psychological
Methods, 5(4), 434-458.).

Even with a relatively low value of k/p (k = number of studies, p = number
of model coefficients), the Type I error rate for testing moderators is
close to nominal as long as one employs something like the Knapp & Hartung
correction. The problem is that power is likely to be very low then as
well. One could try to do a proper power analysis (Hedges, L. V., & Pigott,
T. D. (2004). The power of statistical tests for moderators in
meta-analysis. Psychological Methods, 9(4), 426-445.), but I've never seen
this done in practice (at least not reported). Something like a 'k per p'
rule is of course much easier to implement.

Best,
Wolfgang

-----Original Message-----
From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-
bounces at r-project.org] On Behalf Of Angeline Tsui
Sent: Wednesday, 03 January, 2018 20:07
To: r-sig-meta-analysis at r-project.org
Subject: [R-meta] Question about Cohen's d and meta-regression

Hello all,

I have some questions about effect sizes and meta-regression.

1) I am using between-subject Cohen's d (but this effect size is converted
from a within-subject d based on Morris and Deshan (2002) for my
meta-analysis. The main reason of the conversion is to make this mean
effect size comparable to other meta-analysis in my field, especially when
some other meta-analysis may also report between-subject Cohen's d.
However, I have concerns of interpreting the magnitude of the mean effect
size.

When I interpret the mean Cohen's d, I use the Cohen (1988) rule of thumb:
0.2 -> small, 0.5 -> medium and 0.8 -> large. However, a concern of
applying this Cohen's values rule of thumb to the current meta-analysis is
that it is not "context-specific". For example, a small effect size of 0.3
can be regarded as large effect size in some other contexts.

In this case, what other metrics would you recommend me to use to report
the mean effect size magnitude? I have searched some resources online and
see recommendation about the use of Cohen's U3 index or Common language
effect size statistics (CLES)? From what I understand, both U3 index is a
percentile index whereas CLES is a probability index. But I am not familiar
with these two metrics, can someone give me recommendation which index is
better?

Finally, if I change my effect size to hedge's g because the sample size of
each study is small and I need to correct it by converting Cohen's d to
hedge's g. Can I still convert the mean hedge's g using the same formula of
Cohen's U3 or CLES for interpreting the percentile/probability of the mean
effect size?

2) I ran a meta-regression model with which include a number of moderators.
My concern is the minimum sample size needed for each moderator level. I
was not able to find resources for recommendation of sample size for
moderator analysis. I did learn something from my teacher, she told me that
the rule of thumb is at least 6 studies for a moderator analysis level
(e.g., Gender is a moderator. Then I need to have at least 6 studies that
examined male and at least 6 studies that examined female). Is this true? I
again was not able to find this rule of thumb from books.

Thanks,
Angeline