Skip to content

segfaults from lmer on a 64-bit system

5 messages · Hansen, Ben, Douglas Bates, Ben Hansen

Original
Douglas Bates
# 
I would first try later versions of both the Matrix and lme4 packages.
 Versions currently on CRAN are Matrix_0.999375-11 and
lme4_0.999375-24

If you continue to get segfaults then having a reproducible example,
along with the data - anonymized if you wish, will be important.

You can determine the versions of the packages that you are using with

sessionInfo()
On Wed, Aug 13, 2008 at 12:06 AM, Hansen, Ben <bbh at umich.edu> wrote:
Ben Hansen
# 
Thanks for these suggestions.

I updated to the latest versions of the packages, and the segfault persists.

I have isolated an example for reproduction purposes. The data are given in
    http://www.stat.lsa.umich.edu/~bbh/segfault-bh.rda

I get segfaults when I do:
The segfault occurs on a couple of setups:
1.  64-bit Red Hat linux; R version 2.7.1:
R version 2.7.1 (2008-06-23)
x86_64-unknown-linux-gnu

locale:
LC_CTYPE=en_US.UTF-8;LC_NUMERIC=C;LC_TIME=en_US.UTF-8;LC_COLLATE=en_US.UTF-8
;LC_MONETARY=C;LC_MESSAGES=en_US.UTF-8;LC_PAPER=en_US.UTF-8;LC_NAME=C;LC_ADD
RESS=C;LC_TELEPHONE=C;LC_MEASUREMENT=en_US.UTF-8;LC_IDENTIFICATION=C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
[1] lme4_0.999375-24   Matrix_0.999375-11 lattice_0.17-8

loaded via a namespace (and not attached):
[1] grid_2.7.1  tools_2.7.1

2. Mac OS; R version 2.7.1:
R version 2.7.1 (2008-06-23)
powerpc-apple-darwin8.10.1

locale:
C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
[1] lme4_0.999375-22   Matrix_0.999375-10 lattice_0.17-8

loaded via a namespace (and not attached):
[1] grid_2.7.1

3.  I also got the segfault on the 64-bit linux machine using R version
2.6.1, lme4 version 0.99875-9 (I think), and Matrix version 0.999375-9 (I
think).

4. *On the other hand,* it sort of works (without segfaulting at least)
using older R, lme4 and Matrix on my Mac:
R version 2.6.2 (2008-02-08)
powerpc-apple-darwin8.10.1

locale:
C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
[1] lme4_0.99875-9    Matrix_0.999375-9 lattice_0.17-6

loaded via a namespace (and not attached):
[1] grid_2.6.2
Warning messages:
1: In .local(x, ..., value) :
  Estimated variance-covariance for factor 'by.var' is singular

2: In .local(x, ..., value) :
  nlminb returned message false convergence (8)

--Perhaps the warnings contain a clue as to what the problem is?

Thanks for your consideration of this.

Ben
On 8/13/08 2:55 AM, "Douglas Bates" <bates at stat.wisc.edu> wrote:

            

      
      
      
    

        

      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Douglas Bates
    

    

      
      #
    

  


  

      
Thanks for sending the data and the formula, Ben.

I have good news and bad news.  The good news is that I was able to
get consistent behavior on this example from different systems.  The
bad news is that I did so by having R segfault on a Mac.

I'm not sure exactly where the segfault occurs but the underlying
problem is your random effects specification.  You are specifying a
huge number of random effects (27 terms, some of which may correspond
to more than one random effect) for each level of the grouping factor,
by.var, which has only 4 levels.  You are trying to estimate something
like 380 variances or covariances from 4 groups.

Specifying the random effects is not simply a matter of incorporating
every fixed effect as a random effects and allowing a general
variance-covariance structure.  In this case about the best that you
can hope to include with only 4 levels of the grouping factor is
(1|by.var).

        
On Thu, Aug 14, 2008 at 5:21 AM, Ben Hansen <bbh at umich.edu> wrote:

            

      
      
      
    

        

      
      
    

            

      
      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Hansen, Ben
    

    

      
      #
    

  


  

      
Thanks much for looking into this, Doug. 

I recognize that what I'm doing here may seem odd.  My hope was to adapt
the package to a non-standard purpose of mine.  For this purpose, the
variances of the random effects are not of central interest, and it's OK
if they (those variances) are not particularly well estimated.  I found
encouragement in Gelman and Hill's regression book (\S 12.9, pp.275 ff),
which says it's a mistake to think that multilevel models require a
minimum number of groups.  As they put it, the issue is just that when
the number of groups is small then the random-effects variances will
tend to be overestimated.  For my purposes, that would have been OK. 

You suggest only modeling the intercept as random, given that I've only
got four groups.  For my particular purposes, that isn't an option.
Could lmer be expected to better handle my (admittedly ornate) random
effects specfication if there were more groups than 4 -- say, 6 or 8?

Thanks again for the attention.

Ben

Ben B. Hansen
Assistant Professor
Statistics Department and Institute for Social Research
www.stat.lsa.umich.edu/~bbh/
University of Michigan
Ann Arbor, MI 48109-1107


-----Original Message-----
From: dmbates at gmail.com [mailto:dmbates at gmail.com] On Behalf Of Douglas
Bates
Sent: Thursday, August 14, 2008 5:08 AM
To: Hansen, Ben
Cc: r-sig-mixed-models at r-project.org; Caird, Andrew J
Subject: Re: [R-sig-ME] segfaults from lmer on a 64-bit system

Thanks for sending the data and the formula, Ben.

I have good news and bad news.  The good news is that I was able to
get consistent behavior on this example from different systems.  The
bad news is that I did so by having R segfault on a Mac.

I'm not sure exactly where the segfault occurs but the underlying
problem is your random effects specification.  You are specifying a
huge number of random effects (27 terms, some of which may correspond
to more than one random effect) for each level of the grouping factor,
by.var, which has only 4 levels.  You are trying to estimate something
like 380 variances or covariances from 4 groups.

Specifying the random effects is not simply a matter of incorporating
every fixed effect as a random effects and allowing a general
variance-covariance structure.  In this case about the best that you
can hope to include with only 4 levels of the grouping factor is
(1|by.var).

        
On Thu, Aug 14, 2008 at 5:21 AM, Ben Hansen <bbh at umich.edu> wrote:

            

      
      
      
    

        
persists.

            

      
      
      
    

        
given in

            

      
      
      
    

        
data=mydata)

            

      
      
      
    

        
LC_CTYPE=en_US.UTF-8;LC_NUMERIC=C;LC_TIME=en_US.UTF-8;LC_COLLATE=en_US.U
TF-8

            

      
      
      
    

        
;LC_MONETARY=C;LC_MESSAGES=en_US.UTF-8;LC_PAPER=en_US.UTF-8;LC_NAME=C;LC
_ADD

            

      
      
      
    

        
version

            

      
      
      
    

        
(I

            

      
      
      
    

        
least)

            

      
      
      
    

        
packages.

            

      
      
      
    

        
with

            

      
      
      
    

        
I'm

            

      
      
      
    

        
Using

            

      
      
      
    

        
However,

            

      
      
      
    

        
with

            

      
      
      
    

        
fails

            

      
      
      
    

        
few

            

      
      
      
    

        
variants