[R-meta] Effect sizes for mixed-effects models
Lena, The formula you tried from Hedges 2007 is derived under the assumption that treatment assignment is at the cluster level, so I don't think it will work for your mixed design. The following post might be useful to answer your questions: https://www.jepusto.com/alternative-formulas-for-the-smd/ In it, I suggest a quite general approach to estimating the variance of a standardized mean difference effect size, even if it is based on a complex experimental design. Suppose that you calculate the SMD estimate as d = b / S, where b is the unstandardized mean difference (which in your design involves a combination of within- and between-Ss comparisons) and S is the standard deviation of the outcome, which generally might involve a sum of multiple variance components. A delta-method approximation to the variance of d is Vd = (SEb / S)^2 + d^2 / (2 v), where SEb is the standard error of b, S is the denominator of the effect size estimate, d is the effect size estimate, and v is the degrees of freedom of S^2, defined by v = 2[ E(S^2)]^2 / Var(S^2). The SEb should usually be reported in primary studies (or can be back-calculated from t statistics or CIs). Thus, the only tricky bit is to find the degrees of freedom for the standardizing variance S^2. You might need to just make a rough approximation, based on for instance the total number of participants. Using a rough approximation (e.g., v = 30) should not have much effect on the total estimated variance Vd unless d is very large, so personally I would not worry too much about getting it perfect. As I explain in the post, you can also use the degrees of freedom v to do Hedges' g correction, taking g = J(v) * d, where J(v) = 1 - 3 / (4 * v - 1). Again, it's not worth worrying about getting the degrees of freedom perfect. Consider that J(30) = 0.9748 and J(60) = 0.9874, so the g estimate will differ by only a tiny amount depending on the degrees of freedom you use. James On Sat, Oct 19, 2019 at 2:41 PM Lena Sch?fer <lenaschaefer2304 at gmail.com> wrote:
Hi everyone, I am writing to ask two questions related to the calculation of effect sizes for mixed-effects models for a meta-analysis. To derive effect sizes for mixed-effects models, we generally follow the Hedges 2007 paper ( https://journals.sagepub.com/doi/abs/10.3102/1076998606298043?journalCode=jebb < https://journals.sagepub.com/doi/abs/10.3102/1076998606298043?journalCode=jebb>) and a blogpost by Jake Westfall on effect-size calculations for within-subjects designs ( http://jakewestfall.org/blog/index.php/2016/03/25/five-different-cohens-d-statistics-for-within-subject-designs/ < http://jakewestfall.org/blog/index.php/2016/03/25/five-different-cohens-d-statistics-for-within-subject-designs/
):
1. Variance for complex mixed-effects models
While the calculation of Cohen?s d is unproblematic (formula 8 on page 346
in Hedges, 2007), the calculation of the respective variance turned out to
be difficult for complex study designs. Hedge?s provided the following
formula () to derive V(dw):
V(dw) = ((NT + NC) / (NT * NC)) * ((1+(n-1)p)/(1-p)) + ((dw^2) / (2(N ?
M)))
with NT referring to the number of observations in the treatment group, NC
referring to the number of observations in the control group, N referring
to the total number of observations (NT + NC = N), n referring to the
number of observations per cluster, p referring to the ICC, and M referring
to the number of clusters.
For our meta-analysis, we want to derive the variance related to Cohen?s d
for a mixed-subjects design with some participant conducting a task only in
the control condition and other participants conducting the task in the
control and in the experimental condition (within-subjects design). Since
the number of observations per cluster differs (some participants have 30
observations, others have 60) we decided to use the variance formula for
unequal cluster sample sizes in which n is substituted with the cluster
sample size ? (formula 18 on page 350):
? = ((NC * ?mTi = 1 (nTi)^2) / (NT * N)) + ((NT * ?mCi = 1 (nCi)^2) / (NC
* N))
iWhile we expected that this formula would yield an unequal cluster sample
size between 30 and 60, it gives us a value of 30 (which is equal to the
cluster sample size if this would be a between-subjects design). This
suggests that the formula cannot account for the participants which are
both in the control and the experimental condition. Do you have any advice
on how we could derive an accurate variance estimate for such a design?
2. Turning Cohen?s d into Hedge?s g for mixed-models
Finally, we want to transform Cohen?s d into Hedge?s g using:
g(d) = d * (1- ((3) / (4 * df - 1))
We are uncertain how to best estimate the dfs in our mixed-models. We
considered using Kenward-Roger approximated dfs but this does not seem
feasible since we only have access to parts of the raw data-sets used to
derive dw and V{dw}. Potentially, another option would be to estimate the
dfs via the effective sample size. This seems more feasible since the
authors of primary papers provided us with the ICC related to each model.
What do you think about this option?
If you have any thoughts on this, we would greatly appreciate it if you
could let us know what you think. Thank you for taking the time to consider
our request, and please don?t hesitate to reach out if anything is unclear.
Thank you very much and best regards,
Lena Sch?fer
On behalf of a collaborative team that additionally includes Leah
Somerville (head of the Affective Neuroscience and Development Laboratory),
Katherine Powers (former postdoc in the Affective Neuroscience and
Development Laboratory) and Bernd Figner (Radboud University).
[[alternative HTML version deleted]]
_______________________________________________ R-sig-meta-analysis mailing list R-sig-meta-analysis at r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis