Help with determining effect sizes

Dear Francesco,

I don't think there is a "standard" way to calculate effect sizes for
linear mixed models due to the way the variance is partitioned (see
e.g. [1]).
One way to compute something similar to Cohen's d would be to divide
the difference between the estimated means of two conditions by a
rough estimate of the standard deviation of the response variable
which you can get by
sd(predict(your_model_name))

Best,
Maarten

[1] 
https://afex.singmann.science/forums/topic/compute-effect-sizes-for-mixed-objects#post-295


On Sat, Oct 5, 2019 at 10:01 AM Francesco Romano <
fbromano77 at gmail.com> wrote:

Dear all,

A journal has asked that I determine the effect sizes for a series
of
dummy-coded contrasts from the following ME model:

RT ~ Group * Grammaticality + (1 + Grammaticality | Participant) +
    (1 + Group | item)

Here RT is my continuous outcome variable measured in milliseconds,
Group
is a factor with 3 levels (NS, L2, and HL), and Grammaticality a
factor
with 2 levels (gr and ungr). After relevelling ?NOTE: I am
deliberately
omitting the call for each new relevelled model here? I obtained a
series
of contrasts which are tabulated below (not sure you can view this
whole):


Reference level

Contrasts

Estimate

(ms)

Effect size

(Cohen?s *d*)

SE

df

*t*

*p*

HL

GR vs UNGR

 -213



89

72.13

-2.399

< .05*

L2

GR vs UNGR

 -408



90

74.18

-4.513

< .001***

L1

GR vs UNGR

-111



73

70.02

-1.520

.05

HL > L2

 -25



191

43.48

-.135

.05

GR

L1 > HL

 400



175

43.81

2.286

< .05*



L1 > L2

 374



179

43.59

2.092

< .05*



HL > L2

 -219



179

42.70

-1.226

.05

UNGR

L1 > HL

 298



164

43

1.817

.05

L1> L2

77



166

42.03

.469

.05

How would I go about determining the Cohen's *d* for each of the
contrasts?

The model call is:

Linear mixed model fit by REML. t-tests use Satterthwaite's method
['lmerModLmerTest']
Formula: RT ~ Group * Grammaticality + (1 + Grammaticality |
Participant) +

    (1 + Group | item)
   Data: RTanalysis

REML criterion at convergence: 52800

Scaled residuals:
    Min      1Q  Median      3Q     Max
-2.1696 -0.6536 -0.1654  0.5060  5.0134

Random effects:
 Groups      Name               Variance Std.Dev. Corr
 item        (Intercept)         71442   267.29
             GroupL2              1144    33.82    0.80
             GroupNS              9951    99.76   -0.43 -0.88
 Participant (Intercept)        235216   484.99
             Grammaticalityungr  50740   225.25   -0.39
 Residual                       378074   614.88
Number of obs: 3342, groups:  item, 144; Participant, 46

Fixed effects:
                           Estimate Std. Error      df t value
Pr(>|t|)
(Intercept)                 2801.98     136.70   48.85  20.498   <2
e-16 ***
GroupL2                      -25.86     191.20   43.48  -
0.135   0.8931
GroupNS                     -400.63     175.22   43.81  -
2.286   0.0271 *
Grammaticalityungr          -213.87      89.17   72.13  -
2.399   0.0190 *
GroupL2:Grammaticalityungr  -194.57     107.25   42.55  -
1.814   0.0767 .
GroupNS:Grammaticalityungr   102.31      99.39   43.45   1.029   0.
3090
---
Signif. codes:  0 ?***? 0.001 ?**? 0.01 ?*? 0.05 ?.? 0.1 ? ? 1

Correlation of Fixed Effects:
            (Intr) GropL2 GropNS Grmmtc GrL2:G
GroupL2     -0.672
GroupNS     -0.744  0.526
Grmmtcltyng -0.404  0.222  0.260
GrpL2:Grmmt  0.259 -0.391 -0.205 -0.589
GrpNS:Grmmt  0.299 -0.202 -0.392 -0.702  0.540
convergence code: 0
Model failed to converge with max|grad| = 0.0477764 (tol = 0.002,
component
1)

The distribution of the outcome is fairly normal and the overall
mean,
without considering the two fixed effects, is very close to the
means of
each of the three groups (without considering the effect of
Grammaticality)
as well as the means of the two levels of grammaticality (without
considering the effect of group).

The package simR can simulate data to determine power, amongst
other things,
but I am not sure how to do this for models with interactions such
as mine.

Use of simR is recommended in Brysbaert and Stevens (2018)
https://www.journalofcognition.org/articles/10.5334/joc.10/.
Perhaps there
is a simpler way of extracting *d *from the stats I already know?

Any help would be greatly appreciated,

Francesco

        [[alternative HTML version deleted]]

See the web app by Jake Westfall:
https://jakewestfall.shinyapps.io/crossedpower/

And their JEP:General paper:
http://doi.org/10.1037/xge0000014

If I'm not mistaken, you would standardise the estimates of differences
by the sum of all variances (random intercepts and slopes + residual),
but you'll need to make sure that's the right formula (given your
desgin).

Jo?o

On Sat, 2019-10-05 at 12:13 +0200, Maarten Jung wrote:

Dear Francesco,

I don't think there is a "standard" way to calculate effect sizes for
linear mixed models due to the way the variance is partitioned (see
e.g. [1]).
One way to compute something similar to Cohen's d would be to divide
the difference between the estimated means of two conditions by a
rough estimate of the standard deviation of the response variable
which you can get by
sd(predict(your_model_name))

Best,
Maarten

[1]

https://afex.singmann.science/forums/topic/compute-effect-sizes-for-mixed-objects#post-295

On Sat, Oct 5, 2019 at 10:01 AM Francesco Romano <
fbromano77 at gmail.com> wrote:

Dear all,

A journal has asked that I determine the effect sizes for a series
of
dummy-coded contrasts from the following ME model:

RT ~ Group * Grammaticality + (1 + Grammaticality | Participant) +
    (1 + Group | item)

Here RT is my continuous outcome variable measured in milliseconds,
Group
is a factor with 3 levels (NS, L2, and HL), and Grammaticality a
factor
with 2 levels (gr and ungr). After relevelling ?NOTE: I am
deliberately
omitting the call for each new relevelled model here? I obtained a
series
of contrasts which are tabulated below (not sure you can view this
whole):


Reference level

Contrasts

Estimate

(ms)

Effect size

(Cohen?s *d*)

SE

df

*t*

*p*

HL

GR vs UNGR

 -213



89

72.13

-2.399

< .05*

L2

GR vs UNGR

 -408



90

74.18

-4.513

< .001***

L1

GR vs UNGR

-111



73

70.02

-1.520

.05

HL > L2

 -25



191

43.48

-.135

.05

GR

L1 > HL

 400



175

43.81

2.286

< .05*



L1 > L2

 374



179

43.59

2.092

< .05*



HL > L2

 -219



179

42.70

-1.226

.05

UNGR

L1 > HL

 298



164

43

1.817

.05

L1> L2

77



166

42.03

.469

.05

How would I go about determining the Cohen's *d* for each of the
contrasts?

The model call is:

Linear mixed model fit by REML. t-tests use Satterthwaite's method
['lmerModLmerTest']
Formula: RT ~ Group * Grammaticality + (1 + Grammaticality |
Participant) +

    (1 + Group | item)
   Data: RTanalysis

REML criterion at convergence: 52800

Scaled residuals:
    Min      1Q  Median      3Q     Max
-2.1696 -0.6536 -0.1654  0.5060  5.0134

Random effects:
 Groups      Name               Variance Std.Dev. Corr
 item        (Intercept)         71442   267.29
             GroupL2              1144    33.82    0.80
             GroupNS              9951    99.76   -0.43 -0.88
 Participant (Intercept)        235216   484.99
             Grammaticalityungr  50740   225.25   -0.39
 Residual                       378074   614.88
Number of obs: 3342, groups:  item, 144; Participant, 46

Fixed effects:
                           Estimate Std. Error      df t value
Pr(>|t|)
(Intercept)                 2801.98     136.70   48.85  20.498   <2
e-16 ***
GroupL2                      -25.86     191.20   43.48  -
0.135   0.8931
GroupNS                     -400.63     175.22   43.81  -
2.286   0.0271 *
Grammaticalityungr          -213.87      89.17   72.13  -
2.399   0.0190 *
GroupL2:Grammaticalityungr  -194.57     107.25   42.55  -
1.814   0.0767 .
GroupNS:Grammaticalityungr   102.31      99.39   43.45   1.029   0.
3090
---
Signif. codes:  0 ?***? 0.001 ?**? 0.01 ?*? 0.05 ?.? 0.1 ? ? 1

Correlation of Fixed Effects:
            (Intr) GropL2 GropNS Grmmtc GrL2:G
GroupL2     -0.672
GroupNS     -0.744  0.526
Grmmtcltyng -0.404  0.222  0.260
GrpL2:Grmmt  0.259 -0.391 -0.205 -0.589
GrpNS:Grmmt  0.299 -0.202 -0.392 -0.702  0.540
convergence code: 0
Model failed to converge with max|grad| = 0.0477764 (tol = 0.002,
component
1)

The distribution of the outcome is fairly normal and the overall
mean,
without considering the two fixed effects, is very close to the
means of
each of the three groups (without considering the effect of
Grammaticality)
as well as the means of the two levels of grammaticality (without
considering the effect of group).

The package simR can simulate data to determine power, amongst
other things,
but I am not sure how to do this for models with interactions such
as mine.

Use of simR is recommended in Brysbaert and Stevens (2018)
https://www.journalofcognition.org/articles/10.5334/joc.10/.
Perhaps there
is a simpler way of extracting *d *from the stats I already know?

Any help would be greatly appreciated,

Francesco

        [[alternative HTML version deleted]]