Minimum detectable effect size in linear mixed model
Hi Sacha, Correct me if I'm wrong, but I tend to think this is more like a sensitivity analysis (given alpha, power, and N, solve for the required effect size). If the minimum detectable effect size at 80% power ends up so large that it exceeds the typical range in the field (say, a .6 correlation is the minimum whereas a .2 is typically expected), then we may say the study is underpowered. So I think I made a mistake with question (2) - the MDES should be compared to an effect size with practical importance, not the observed effect size. Han
On Sat, Jul 4, 2020 at 12:07 PM varin sacha <varinsacha at yahoo.fr> wrote:
Hi, Is the question about post hoc power analysis ? Post hoc power analyses are usually not suggested. (See for example The abuse of power...hoenig & heisey). You should do an a priori power analysis. If you then do the small sample study and obtain a negative result, you have no idea why ? you are stuck. That is why I always tell people not to do a study where everything rides on a significant result. It is an unnecessary gamble. It is always better to realize an a priori power analysis to know Type II error and the power in case of the test is not significant. Also, it is very easy to, a priori, estimate the power of say, a medium, effect size. So there is little reason for not doing that at the beginning. Best, Sacha Envoy? de mon iPhone
Le 4 juil. 2020 ? 01:04, Patrick (Malone Quantitative) <
malone at malonequantitative.com> a ?crit :
?No, because I don't think it can be. That's not how power analysis
works.
It's bad practice.
On Fri, Jul 3, 2020, 6:42 PM Han Zhang <hanzh at umich.edu> wrote: Hi Pat, Thanks for your quick reply. Yes, I already have the data and the actual effects, and the analysis was suggested by a reviewer. Can you
elaborate on
when do you think such an analysis might be justified? Thanks! Han On Fri, Jul 3, 2020 at 6:34 PM Patrick (Malone Quantitative) < malone at malonequantitative.com> wrote:
Han, (1) Usually, yes, but . . . (2) If you have an actual effect, does that mean you're doing post hoc power analysis? If so, that's a whole can of worms, for which the best advice I have is "don't do it." Use the size of the confidence interval of your estimate as an assessment of sample adequacy. Pat On Fri, Jul 3, 2020 at 6:27 PM Han Zhang <hanzh at umich.edu> wrote:
Hello, I'm trying to find the minimum detectable effect size (MDES) given my sample, alpha (.05), and desired power (90%) in a linear mixed model setting. I'm using the simr package for a simulation-based approach.
What I
did is changing the original effect size to a series of hypothetical
effect
sizes and find the minimum effect size that has a 90% chance of
producing a
significant result. Below is a toy code:
library(lmerTest)
library(simr)
# fit the model
model <- lmer(Reaction ~ Days + (Days | Subject), sleepstudy)
summary(model)
Fixed effects:
Estimate Std. Error df t value Pr(>|t|)
(Intercept) 251.405 6.825 17.000 36.838 < 2e-16 ***
Days 10.467 1.546 17.000 6.771 3.26e-06 ***
Here is the code for minimum detectable effect size:
pwr <- NA
# define a set of reasonable effect sizes
es <- seq(0, 10, 2)
# loop through the effect sizes
for (i in 1:length(es)) {
# replace the original effect size with new one
fixef(model)['Days'] = es[i]
# run simulation to obtain power estimate
pwr.summary <- summary(powerSim(
model,
test = fixed('Days', "t"),
nsim = 100,
progress = T
))
# store output
pwr[i] <- as.numeric(pwr.summary)[3]
}
# display results
cbind("Coefficient" = es,
Power = pwr)
Output:
Coefficient Power
[1,] 0 0.09
[2,] 2 0.24
[3,] 4 0.60
[4,] 6 0.99
[5,] 8 1.00
[6,] 10 1.00
My questions:
(1) Is this the right way to find the MDES?
(2) I have some trouble making sense of the output. Can I say the
following: because the estimated power when the effect = 6 is 99%, and
because the actual model has an estimate of 10.47, then the study is
sufficiently powered? Conversely, imagine that if the actual estimate
was
3.0, then can I say the study is insufficiently powered? Thank you, Han -- Han Zhang, Ph.D. Department of Psychology University of Michigan, Ann Arbor https://sites.lsa.umich.edu/hanzh/ [[alternative HTML version deleted]]
_______________________________________________ R-sig-mixed-models at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
-- Patrick S. Malone, Ph.D., Malone Quantitative NEW Service Models: http://malonequantitative.com He/Him/His
-- Han Zhang, Ph.D. Department of Psychology University of Michigan, Ann Arbor https://sites.lsa.umich.edu/hanzh/
[[alternative HTML version deleted]]
_______________________________________________ R-sig-mixed-models at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
Han Zhang, Ph.D. Department of Psychology University of Michigan, Ann Arbor https://sites.lsa.umich.edu/hanzh/ [[alternative HTML version deleted]]