Skip to content

Effect sizes for mixed-effects models

5 messages · David Duffy, Phillip Alday, Schäfer, L. (Lena)

#
Hi everyone,
I am writing to ask two questions related to the calculation of effect sizes for mixed-effects models for a meta-analysis.
To derive effect sizes for mixed-effects models, we generally follow the Hedges 2007 paper (https://journals.sagepub.com/doi/abs/10.3102/1076998606298043?journalCode=jebb) and a blogpost by Jake Westfall on effect-size calculations for within-subjects designs (http://jakewestfall.org/blog/index.php/2016/03/25/five-different-cohens-d-statistics-for-within-subject-designs/):
1.     Variance for complex mixed-effects models
While the calculation of Cohen?s d is unproblematic (formula 8 on page 346 in Hedges, 2007), the calculation of the respective variance turned out to be difficult for complex study designs. Hedge?s provided the following formula () to derive V(dw):
V(dw) = ((NT + NC) / (NT * NC)) * ((1+(n-1)p)/(1-p)) + ((dw^2) / (2(N ? M)))
with NT referring to the number of observations in the treatment group, NC referring to the number of observations in the control group, N referring to the total number of observations (NT + NC  = N), n referring to the number of observations per cluster, p referring to the ICC, and M referring to the number of clusters.
 For our meta-analysis, we want to derive the variance related to Cohen?s d for a mixed-subjects design with some participant conducting a task only in the control condition and other participants conducting the task in the control and in the experimental condition (within-subjects design). Since the number of observations per cluster differs (some participants have 30 observations, others have 60) we decided to use the variance formula for unequal cluster sample sizes in which n is substituted with the cluster sample size ? (formula 18 on page 350):
? = ((NC * ?mTi = 1 (nTi)^2) / (NT * N)) +  ((NT * ?mCi = 1 (nCi)^2) / (NC * N))
iWhile we expected that this formula would yield an unequal cluster sample size between 30 and 60, it gives us a value of 30 (which is equal to the cluster sample size if this would be a between-subjects design). This suggests that the formula cannot account for the participants which are both in the control and the experimental condition. Do you have any advice on how we could derive an accurate variance estimate for such a design?
2. Turning Cohen?s d into Hedge?s g for mixed-models
Finally, we want to transform Cohen?s d into Hedge?s g using:
 g(d) = d * (1- ((3) / (4 * df - 1))
We are uncertain how to best estimate the dfs in our mixed-models. We considered using Kenward-Roger approximated dfs but this does not seem feasible since we only have access to parts of the raw data-sets used to derive dw and V{dw}. Potentially, another option would be to estimate the dfs via the effective sample size. This seems more feasible since the authors of primary papers provided us with the ICC related to each model. What do you think about this option?
If you have any thoughts on this, we would greatly appreciate it if you could let us know what you think. Thank you for taking the time to consider our request, and please don?t hesitate to reach out if anything is unclear.
Thank you very much and best regards,
Lena Sch?fer
On behalf of a collaborative team that additionally includes Leah Somerville (head of the Affective Neuroscience and Development Laboratory), Katherine Powers (former postdoc in the Affective Neuroscience and Development Laboratory) and Bernd Figner (Radboud University).
#
Do you have the raw data or appropriate summary statistics from this mixed-subjects study? Could you get it? I think the "usual" approach would be to use a conservative lower bound for the effective sample size.

Cheers, David Duffy.
#
On 10/10/2019 23:41, Sch?fer, L. (Lena) wrote:
You have 30-60 observations per participant, but how many participants?
If it's the typical 20+ in psychology, I would use the easiest
approximation of all for denominator degrees of freedom: treat them as
infinite, i.e. treat the t values as z values. The Kenward-Roger
approximation really doesn't really change your results for non trivial
datasets and the implementation in R (in pbkrtest, which lmerTest and
car::Anova() use internally) computes a matrix inverse for an n x n
matrix, where n is the total number of observations. This is
computationally painful for non trivial n with minimal benefit.

Best (from next door at the MPI),

Phillip
#
Hi Phillip, 

Thank you for your response; this was very helpful. The studies analyzed using mixed-effects models have a minimum of 24 participants. Setting the degrees of freedom equal to infinity (i.e., Cohen?s d = Hedge?s g) seems to be justifiable in that case. 

Thanks, 
Lena
#
Hi David,

Thank you for your response! We have the raw data for some studies but only info on the sample/cluster size, the lme4 output and the respective ICC for other studies. Since we are conducting a meta-analysis, we cannot get access to all data-sets (eg., problems with data protection, unpublished data-sets). However, using the info on the sample and cluster size and the ICC, we can derive the effective sample size (Aarts, Verhage, Veenvliet, Dolan, & van der Sluis, 2014) related to the study.

I am not sure what you are referring to with *lower bound for the effective sample size*. Is this value calculated using another formula or is it essentially equal to the number of participants (as suggested by Phillip Alday)?

Thank you for a clarification!

Best,
Lena



Am 10.10.2019 um 19:34 schrieb David Duffy <David.Duffy at qimrberghofer.edu.au<mailto:David.Duffy at qimrberghofer.edu.au>>:


we want to derive the variance related to Cohen?s d for a mixed-subjects design with some participant conducting a task
only in the control condition and other participants conducting the task in the control and in the experimental condition
(within-subjects design).

[...] we only have access to parts of the raw data-sets

Do you have the raw data or appropriate summary statistics from this mixed-subjects study? Could you get it? I think the "usual" approach would be to use a conservative lower bound for the effective sample size.

Cheers, David Duffy.