test significance of single random effect
1. Re: test significance of single random effect
(Achaz von Hardenberg)
2. Re: test significance of single random effect (Ben Bolker)
----------------------------------------------------------------------
Message: 1
Date: Sat, 28 Nov 2009 15:23:01 +0000
From: Achaz von Hardenberg <achaz.hardenberg at gmail.com>
Subject: Re: [R-sig-ME] test significance of single random effect
To: R Mixed Models <r-sig-mixed-models at r-project.org>
Message-ID: <E7D9F7E1-84DA-4C1F-B9D4-F6222564975E at pngp.it>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Dear all,
I am coming back on the recent issue on how to test the significance
of a single random term in linear mixed models...
In Zuur et al. "Mixed Models and Extentions in Ecology with R"
Springer, 2009, the authors suggest to compare a lme model (with the
random effect) with a gls model with the same fixed effects structure,
and then compare the AICs of the two models or using a likelihood
ratio test via the ANOVA comand (pages 122 - 128).
See also Verbeke and Molenberghs (2000)... The correction for testing on the boundary is described on page 123 (and see also V&M) and is viewed as quick and dirty. My question to you...why do you want to test the significance of a random term in a linear mixed model? Why not include it purely based on the design of the experiment? Alain
I would be interested in hearing the opinion of other members of the list on this approach... Thanks a lot, Achaz On 17 Nov 2009, at 20:41, Tom Van Dooren wrote:
With REML=FALSE RLRsim seems to work fine in R 2.10, if I use the
design matrix and Zt as arguments in LRTSim().
Otherwise I didn't get useful results out.
That's not too much of a problem.
It is not difficult to simulate the null model without random
effect, extract logLikelihoods from the (generalized) mixed model
and the (generalized) linear model fitted to those pseudo-data, to
calculate a distribution of likelihood ratios,
which are then maybe off by a constant.
What I was mainly uncertain about, is whether the log-likelihood of
a mixed model (also fitted to data simulated from the null model
without random effect),
can be used as a statistic itself?
The answer might be a simple NO! of course, or something more
involved...
Tom
Douglas Bates wrote:
On Tue, Nov 17, 2009 at 3:49 AM, Matthias Gralle
<matthias_gralle at eva.mpg.de> wrote:
I had basically the same problem a short time ago, and resorted to
lme
instead of lmer, because one can directly compare lme and lm
objects using
anova(). Is that OK, or is this feature of lme depreciated ?
Is that not possible for linear mixed-effects models fit by lmer
using
REML = FALSE? (Occasionally I lose track of what can be done in
different versions of lme4.) You don't want to compare an lmer model
fit by REML with the log-likelihood of an lm model but you should be
able to compare likelihoods (subject to the caveat that the p-value
for the likelihood ratio test on the boundary of the parameter space
is conservative).
Ben Bolker wrote:
Have you tried the RLRsim package??
Tom Van Dooren wrote:
I tried to find an easy way to test whether the random effect
would be
significant in a (generalized) mixed model with a single random
effect.
It annoyed me that log-likelihoods of lm or glm and lmer are not
necesarily directly comparable -> trouble with calculating
likelihood
ratios.
What do members of this list think of the following simulation
approach?
It basically amounts to simulating a distribution for the log
likelihood,
given the null hypothesis that there is no random effect
variance and that
the fixed effect model is correct.
library(lme4)
mm1 <- lmer(Reaction ~ Days + (1|Subject), sleepstudy)
lm1<- lm(Reaction ~ Days, sleepstudy)
LL<-numeric(500)
for(i in 1:500){
resp<-simulate(lm1)
LL[i]<-logLik(lmer(resp[,1] ~ Days + (1|Subject), sleepstudy))
}
hist(LL)
logLik(mm1)
mean(LL>logLik(mm1))
_______________________________________________ R-sig-mixed-models at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
-- Matthias Gralle, PhD Dept. Evolutionary Genetics Max Planck Institute for Evolutionary Anthropology Deutscher Platz 6 04103 Leipzig, Germany Tel +49 341 3550 519 Fax +49 341 3550 555
_______________________________________________ R-sig-mixed-models at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
_______________________________________________ R-sig-mixed-models at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
Dr. Achaz von Hardenberg -------------------------------------------------------------------------------------------------------- Centro Studi Fauna Alpina - Alpine Wildlife Research Centre Servizio Sanitario e della Ricerca Scientifica Parco Nazionale Gran Paradiso, Degioz, 11, 11010-Valsavarenche (Ao), Italy Present address: National Centre for Statistical Ecology School of Mathematics, Statistics and Actuarial Science, University of Kent, Canterbury, UK ------------------------------ Message: 2 Date: Sat, 28 Nov 2009 21:49:34 -0500 From: Ben Bolker <bolker at ufl.edu> Subject: Re: [R-sig-ME] test significance of single random effect To: Achaz von Hardenberg <achaz.hardenberg at gmail.com> Cc: R Mixed Models <r-sig-mixed-models at r-project.org> Message-ID: <4B11E13E.3030105 at ufl.edu> Content-Type: text/plain; charset=ISO-8859-1 I think it will be conservative (in the sense of underestimating the significance of the random effect), because of the well-known(?) boundary issue (the null hypothesis for random effects, variance==0, is on the boundary of the feasible space). I went a little overboard in testing this: see <http://glmm.wikidot.com/random-effects-testing> , and feel free to improve it ... Achaz von Hardenberg wrote:
Dear all,
I am coming back on the recent issue on how to test the significance
of a single random term in linear mixed models...
In Zuur et al. "Mixed Models and Extentions in Ecology with R"
Springer, 2009, the authors suggest to compare a lme model (with the
random effect) with a gls model with the same fixed effects structure,
and then compare the AICs of the two models or using a likelihood
ratio test via the ANOVA comand (pages 122 - 128).
I would be interested in hearing the opinion of other members of the
list on this approach...
Thanks a lot,
Achaz
On 17 Nov 2009, at 20:41, Tom Van Dooren wrote:
With REML=FALSE RLRsim seems to work fine in R 2.10, if I use the
design matrix and Zt as arguments in LRTSim().
Otherwise I didn't get useful results out.
That's not too much of a problem.
It is not difficult to simulate the null model without random
effect, extract logLikelihoods from the (generalized) mixed model
and the (generalized) linear model fitted to those pseudo-data, to
calculate a distribution of likelihood ratios,
which are then maybe off by a constant.
What I was mainly uncertain about, is whether the log-likelihood of
a mixed model (also fitted to data simulated from the null model
without random effect),
can be used as a statistic itself?
The answer might be a simple NO! of course, or something more
involved...
Tom
Douglas Bates wrote:
On Tue, Nov 17, 2009 at 3:49 AM, Matthias Gralle
<matthias_gralle at eva.mpg.de> wrote:
I had basically the same problem a short time ago, and resorted to
lme
instead of lmer, because one can directly compare lme and lm
objects using
anova(). Is that OK, or is this feature of lme depreciated ?
Is that not possible for linear mixed-effects models fit by lmer
using
REML = FALSE? (Occasionally I lose track of what can be done in
different versions of lme4.) You don't want to compare an lmer model
fit by REML with the log-likelihood of an lm model but you should be
able to compare likelihoods (subject to the caveat that the p-value
for the likelihood ratio test on the boundary of the parameter space
is conservative).
Ben Bolker wrote:
Have you tried the RLRsim package??
Tom Van Dooren wrote:
I tried to find an easy way to test whether the random effect
would be
significant in a (generalized) mixed model with a single random
effect.
It annoyed me that log-likelihoods of lm or glm and lmer are not
necesarily directly comparable -> trouble with calculating
likelihood
ratios.
What do members of this list think of the following simulation
approach?
It basically amounts to simulating a distribution for the log
likelihood,
given the null hypothesis that there is no random effect
variance and that
the fixed effect model is correct.
library(lme4)
mm1 <- lmer(Reaction ~ Days + (1|Subject), sleepstudy)
lm1<- lm(Reaction ~ Days, sleepstudy)
LL<-numeric(500)
for(i in 1:500){
resp<-simulate(lm1)
LL[i]<-logLik(lmer(resp[,1] ~ Days + (1|Subject), sleepstudy))
}
hist(LL)
logLik(mm1)
mean(LL>logLik(mm1))
_______________________________________________ R-sig-mixed-models at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
-- Matthias Gralle, PhD Dept. Evolutionary Genetics Max Planck Institute for Evolutionary Anthropology Deutscher Platz 6 04103 Leipzig, Germany Tel +49 341 3550 519 Fax +49 341 3550 555
_______________________________________________ R-sig-mixed-models at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
_______________________________________________ R-sig-mixed-models at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
Dr. Achaz von Hardenberg -------------------------------------------------------------------------------------------------------- Centro Studi Fauna Alpina - Alpine Wildlife Research Centre Servizio Sanitario e della Ricerca Scientifica Parco Nazionale Gran Paradiso, Degioz, 11, 11010-Valsavarenche (Ao), Italy Present address: National Centre for Statistical Ecology School of Mathematics, Statistics and Actuarial Science, University of Kent, Canterbury, UK
_______________________________________________ R-sig-mixed-models at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
Dr. Alain F. Zuur First author of: 1. Analysing Ecological Data (2007). Zuur, AF, Ieno, EN and Smith, GM. Springer. 680 p. URL: www.springer.com/0-387-45967-7 2. Mixed effects models and extensions in ecology with R. (2009). Zuur, AF, Ieno, EN, Walker, N, Saveliev, AA, and Smith, GM. Springer. http://www.springer.com/life+sci/ecology/book/978-0-387-87457-9 3. A Beginner's Guide to R (2009). Zuur, AF, Ieno, EN, Meesters, EHWG. Springer http://www.springer.com/statistics/computational/book/978-0-387-93836-3 Other books: http://www.highstat.com/books.htm Statistical consultancy, courses, data analysis and software Highland Statistics Ltd. 6 Laverock road UK - AB41 6FN Newburgh Tel: 0044 1358 788177 Email: highstat at highstat.com URL: www.highstat.com URL: www.brodgar.com