Skip to content

test significance of single random effect

8 messages · Ben Bolker, Matthias Gralle, Douglas Bates +2 more

Original
Tom Van Dooren
# 
I tried to find an easy way to test whether the random effect would be 
significant in a (generalized) mixed model with a single random effect.
It annoyed me that log-likelihoods of lm or glm and lmer are not 
necesarily directly comparable -> trouble with calculating likelihood 
ratios.
What do members of this list think of the following simulation approach?
It basically amounts to simulating a distribution for the log 
likelihood, given the null hypothesis that there is no random effect 
variance and that the fixed effect model is correct.


library(lme4)
mm1 <- lmer(Reaction ~ Days + (1|Subject), sleepstudy)
lm1<- lm(Reaction ~ Days, sleepstudy)


LL<-numeric(500)
for(i in 1:500){
resp<-simulate(lm1)
LL[i]<-logLik(lmer(resp[,1] ~ Days + (1|Subject), sleepstudy))
}

hist(LL)
logLik(mm1)
mean(LL>logLik(mm1))
Ben Bolker
# 
Have you tried the RLRsim package??
Tom Van Dooren wrote:

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Tom Van Dooren
    

    

      
      #
    

  


  

      
Hi Ben,
yes I did. The Orthodont example in the LRTSim() help file ran  
perfectly well using lme(), but not with lmer().
Do you think it is OK to use simulated log-likelihoods as a test  
statistic, instead of a likelihood ratio?
Cheers, Tom


Quoting Ben Bolker <bolker at ufl.edu>:

            

      
      
      
    

        

      
      
    

            

      
      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Matthias Gralle
    

    

      
      #
    

  


  

      
I had basically the same problem a short time ago, and resorted to lme 
instead of lmer, because one can directly compare lme and lm objects 
using anova(). Is that OK, or is this feature of lme depreciated ?

        
Ben Bolker wrote:

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Douglas Bates
    

    

      
      #
    

  


  

      
On Tue, Nov 17, 2009 at 3:49 AM, Matthias Gralle

        
<matthias_gralle at eva.mpg.de> wrote:

            

      
      
      
    

        
Is that not possible for linear mixed-effects models fit by lmer using
REML = FALSE? (Occasionally I lose track of what can be done in
different versions of lme4.)  You don't want to compare an lmer model
fit by REML with the log-likelihood of an lm model but you should be
able to compare likelihoods (subject to the caveat that the p-value
for the likelihood ratio test on the boundary of the parameter space
is conservative).

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Tom Van Dooren
    

    

      
      #
    

  


  

      
With REML=FALSE RLRsim seems to work fine in R 2.10, if I use the design 
matrix and Zt as arguments in LRTSim().
Otherwise I didn't get useful results out.

That's not too much of a problem.
It is not difficult to simulate the null model without random effect, 
extract logLikelihoods from the (generalized) mixed model and the 
(generalized) linear model fitted to those pseudo-data, to calculate a 
distribution of likelihood ratios,
which are then maybe off by a constant.
What I was mainly uncertain about, is whether the log-likelihood of a 
mixed model (also fitted to data simulated from the null model without 
random effect),
can be used as a statistic itself?
The answer might be a simple NO! of course, or something more involved...

Tom

        
Douglas Bates wrote:

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

  
  


  


      

    

    
10 days later

    

      
      

        


  

        

      Achaz von Hardenberg
    

    

      
      #
    

  


  

      
Dear all,
I am coming back on the recent issue on how to test the significance  
of a single random term in linear mixed models...

In Zuur et al.  "Mixed Models and Extentions in Ecology with R"  
Springer, 2009, the authors suggest to compare a lme model (with the  
random effect) with a gls model with the same fixed effects structure,  
and then compare the AICs of the two models or using a likelihood  
ratio test via the ANOVA comand (pages 122 - 128).

I would be interested in hearing the opinion of other members of the  
list on this approach...

Thanks a lot,

Achaz

        
On 17 Nov 2009, at 20:41, Tom Van Dooren wrote:

        

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

        
Dr. Achaz von Hardenberg
--------------------------------------------------------------------------------------------------------
Centro Studi Fauna Alpina - Alpine Wildlife Research Centre
Servizio Sanitario e della Ricerca Scientifica
Parco Nazionale Gran Paradiso, Degioz, 11, 11010-Valsavarenche (Ao),  
Italy

Present address:
National Centre for Statistical Ecology
School of Mathematics, Statistics and Actuarial Science,
University of Kent,  Canterbury, UK

  
  


  


      

    

    

      
      

        


  

        

      Ben Bolker
    

    

      
      #
    

  


  

      
I think it will be conservative (in the sense of underestimating the
significance of the random effect), because of the well-known(?)
boundary issue (the null hypothesis for random effects, variance==0, is
on the boundary of the feasible space).

  I went a little overboard in testing this: see
<http://glmm.wikidot.com/random-effects-testing> , and feel free to
improve it ...

        
Achaz von Hardenberg wrote: