Skip to content

How to assess significance of random effect in lme4

3 messages · Doran, Harold, Spencer Graves

Original
Doran, Harold
# 
These are the posterior variances of the random effects (I think more
properly termed "empirical" posteriors).  Your model apparently includes
three levels of random variation (commu, bcohort, residual). The first
are the variances associated with your commu random effect and the
second are the variances associated with the bcohort random effect.

Accessing either one would require

fm at bVar$commu or fm at bVar$bcohort

Obviously, replace "fm" with the name of your fitted model.

-----Original Message-----
From: r-help-bounces at stat.math.ethz.ch
[mailto:r-help-bounces at stat.math.ethz.ch] On Behalf Of Shige Song
Sent: Wednesday, August 17, 2005 7:50 AM
To: r-help at stat.math.ethz.ch
Subject: Re: [R] How to assess significance of random effect in lme4

Hi Harold,

Thanks for the reply. I looked at my outputs using str() as you
suggested, here is the part you mentioned:

  ..@ bVar     :List of 2
  .. ..$ commu  : num [1, 1, 1:29] 5e-10 5e-10 5e-10 5e-10 5e-10 ...
  .. ..$ bcohort: num [1, 1, 1:6] 1.05e-05 7.45e-06 6.53e-06 8.25e-06
7.11e-06 ...

where commu and bcohort are the two second-level units. Are these
standard errors? Why the second vector contains a series of different
numbers?

Thanks!

Shige
On 8/17/05, Doran, Harold <HDoran at air.org> wrote:
summary().
str().
lme4
______________________________________________
R-help at stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide!
http://www.R-project.org/posting-guide.html
Spencer Graves
# 
Is there some reason you are NOT using "anova", as in "Examples" 
section of "?lmer"?

	  Permit me to summarize what I know about this, and I'll be pleased if 
someone else who thinks they know different would kindly enlighten me 
and others who might otherwise be misled if anything I say is 
inconsistent with the best literature available at the moment:

	  1.  Doug Bates in his PhD dissertation and later in his book with Don 
Watts (1988) Nonlinear Regression Analysis and Its Applications (Wiley) 
split approximation errors in nonlinear least squares into "intrinsic 
curvature" and "parameter effects curvature".  He quantified these two 
problems in the context of roughly three dozen published examples, if my 
memory is correct, and found that in not quite all cases, the parameter 
effects were at least an order of magnitude greater than the intrinsic 
curvature.

	  2.  In nonnormal situations, maximum likelihood is subject to more 
approximation error -- intrinsic curvature -- than "simple" nonlinear 
least squares.  However, I would expect this comparison to still be 
fairly accurate, even if the differences may not be quite as stark.

	  3.  The traditional use of "standard errors" to judge statistical 
significance is subject to both intrinsic and parameter effects errors, 
while likelihood ratio procedures such as anova are subject only to the 
intrinsic curvature (assuming there are no substantive problems with 
nonconvergence).  Consequently, to judge statistical significance of an 
effect, anova is usually substantially better than the so-called Wald 
procedure using approximate standard errors, and is almost never worse. 
  If anyone knows of a case where this is NOT true, I'd like to know.

	  4.  With parameters at a boundary as with variance components, the 
best procedure seems to double the p-value from a nested anova (unless 
the reported p-value is already large).  This is because the 
2*log(likelihood ratio) in such cases is roughly a 50-50 mixture of 0 
and chi-square(1) [if testing only 1 variance component parameter]. 
This is supported by a substantial amount of research, including 
simulations discussed in a chapter in Pinheiro and Bates (2000) 
Mixed-Effects Models in S and S-Plus (Springer).  The may be more 
accurate procedures available in the literature, but none so simple as 
this as far as I know.

	  Comments?
	  spencer graves
p.s.  It looks like fm at bVars is a list containing vectors of length 29 
and 6 in your example.  I don't know what they are, but I don't see how 
they can be standard errors in the usual sense.
Doran, Harold wrote:

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

        

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Spencer Graves
    

    

      
      #
    

  


  

      
I don't have Pinheiro and Bates handy, but as I recall, it's somthing 
like chapter 3.  See "simulate.lme" in package nlme;  with luck, it will 
also be in the index to Pinheiro and Bates.  If not, it's still not hard 
to find.  Their chapter on this includes some marvelous figures 
presenting simulation results presumably produced with a version of 
"simulate.lme".

	  The paper that seemed most insightful to me is Crainiceanu, Ruppert 
and Vogelsang (2003) Some Properties of Likelihood Ratio Tests in Linear 
Mixed Models" Cornell U. 
(http://www.orie.cornell.edu/~davidr/papers/zeroprob_rev01.pdf). 
However, I suggest you also examine Ota et al. (2000) "Approximate 
Likelihood Ratio Tests and Marginal Distributions for Evolutionary Tree 
Models with Constraints on Parameters", Molecular Biology and Evolution 
17:798-803.

	  Just now I found other references by Googling for "likelihood ratio 
with parameter at a boundary".  My literature search on this issue is 
far from exhaustive, and I would be pleased to hear from others who have 
other articles to recommend.

	  spencer graves

#####################

        
Andrew Robinson wrote:

        
> Hi Spencer,
 >
 > thanks for this note to the r-users list.  I wonder if I could ask ...
 >
 >
 >>	  4.  With parameters at a boundary as with variance components, the
 >>best procedure seems to double the p-value from a nested anova (unless
 >>the reported p-value is already large).  This is because the
 >>2*log(likelihood ratio) in such cases is roughly a 50-50 mixture of 0
 >>and chi-square(1) [if testing only 1 variance component parameter].
 >>This is supported by a substantial amount of research, including
 >>simulations discussed in a chapter in Pinheiro and Bates (2000)
 >>Mixed-Effects Models in S and S-Plus (Springer).  The may be more
 >>accurate procedures available in the literature, but none so simple as
 >>this as far as I know.
 >
 >
 > Could you provide a page reference or references in the literature?
 > I've heard similar things before, but never seen them myself.  Don't
 > worry if they're not immediately available.
 >
 > Thanks,
 >
 > Andrew
######################################
	  Is there some reason you are NOT using "anova", as in "Examples"
section of "?lmer"?

	  Permit me to summarize what I know about this, and I'll be pleased if
someone else who thinks they know different would kindly enlighten me
and others who might otherwise be misled if anything I say is
inconsistent with the best literature available at the moment:

	  1.  Doug Bates in his PhD dissertation and later in his book with Don
Watts (1988) Nonlinear Regression Analysis and Its Applications (Wiley)
split approximation errors in nonlinear least squares into "intrinsic
curvature" and "parameter effects curvature".  He quantified these two
problems in the context of roughly three dozen published examples, if my
memory is correct, and found that in not quite all cases, the parameter
effects were at least an order of magnitude greater than the intrinsic
curvature.

	  2.  In nonnormal situations, maximum likelihood is subject to more
approximation error -- intrinsic curvature -- than "simple" nonlinear
least squares.  However, I would expect this comparison to still be
fairly accurate, even if the differences may not be quite as stark.

	  3.  The traditional use of "standard errors" to judge statistical
significance is subject to both intrinsic and parameter effects errors,
while likelihood ratio procedures such as anova are subject only to the
intrinsic curvature (assuming there are no substantive problems with
nonconvergence).  Consequently, to judge statistical significance of an
effect, anova is usually substantially better than the so-called Wald
procedure using approximate standard errors, and is almost never worse.
  If anyone knows of a case where this is NOT true, I'd like to know.

	  4.  With parameters at a boundary as with variance components, the
best procedure seems to double the p-value from a nested anova (unless
the reported p-value is already large).  This is because the
2*log(likelihood ratio) in such cases is roughly a 50-50 mixture of 0
and chi-square(1) [if testing only 1 variance component parameter].
This is supported by a substantial amount of research, including
simulations discussed in a chapter in Pinheiro and Bates (2000)
Mixed-Effects Models in S and S-Plus (Springer).  The may be more
accurate procedures available in the literature, but none so simple as
this as far as I know.

	  Comments?
	  spencer graves
p.s.  It looks like fm at bVars is a list containing vectors of length 29
and 6 in your example.  I don't know what they are, but I don't see how
they can be standard errors in the usual sense.

        
Doran, Harold wrote: