segfaults from lmer on a 64-bit system
Thanks much for looking into this, Doug. I recognize that what I'm doing here may seem odd. My hope was to adapt the package to a non-standard purpose of mine. For this purpose, the variances of the random effects are not of central interest, and it's OK if they (those variances) are not particularly well estimated. I found encouragement in Gelman and Hill's regression book (\S 12.9, pp.275 ff), which says it's a mistake to think that multilevel models require a minimum number of groups. As they put it, the issue is just that when the number of groups is small then the random-effects variances will tend to be overestimated. For my purposes, that would have been OK. You suggest only modeling the intercept as random, given that I've only got four groups. For my particular purposes, that isn't an option. Could lmer be expected to better handle my (admittedly ornate) random effects specfication if there were more groups than 4 -- say, 6 or 8? Thanks again for the attention. Ben Ben B. Hansen Assistant Professor Statistics Department and Institute for Social Research www.stat.lsa.umich.edu/~bbh/ University of Michigan Ann Arbor, MI 48109-1107 -----Original Message----- From: dmbates at gmail.com [mailto:dmbates at gmail.com] On Behalf Of Douglas Bates Sent: Thursday, August 14, 2008 5:08 AM To: Hansen, Ben Cc: r-sig-mixed-models at r-project.org; Caird, Andrew J Subject: Re: [R-sig-ME] segfaults from lmer on a 64-bit system Thanks for sending the data and the formula, Ben. I have good news and bad news. The good news is that I was able to get consistent behavior on this example from different systems. The bad news is that I did so by having R segfault on a Mac. I'm not sure exactly where the segfault occurs but the underlying problem is your random effects specification. You are specifying a huge number of random effects (27 terms, some of which may correspond to more than one random effect) for each level of the grouping factor, by.var, which has only 4 levels. You are trying to estimate something like 380 variances or covariances from 4 groups. Specifying the random effects is not simply a matter of incorporating every fixed effect as a random effects and allowing a general variance-covariance structure. In this case about the best that you can hope to include with only 4 levels of the grouping factor is (1|by.var).
On Thu, Aug 14, 2008 at 5:21 AM, Ben Hansen <bbh at umich.edu> wrote:
Thanks for these suggestions. I updated to the latest versions of the packages, and the segfault
persists.
I have isolated an example for reproduction purposes. The data are
given in
http://www.stat.lsa.umich.edu/~bbh/segfault-bh.rda I get segfaults when I do:
library(lme4) ; load("segfault-bh.rda") ; mylme <-lmer(myfmla,
data=mydata)
The segfault occurs on a couple of setups: 1. 64-bit Red Hat linux; R version 2.7.1:
sessionInfo()
R version 2.7.1 (2008-06-23) x86_64-unknown-linux-gnu locale:
LC_CTYPE=en_US.UTF-8;LC_NUMERIC=C;LC_TIME=en_US.UTF-8;LC_COLLATE=en_US.U TF-8
;LC_MONETARY=C;LC_MESSAGES=en_US.UTF-8;LC_PAPER=en_US.UTF-8;LC_NAME=C;LC _ADD
RESS=C;LC_TELEPHONE=C;LC_MEASUREMENT=en_US.UTF-8;LC_IDENTIFICATION=C attached base packages: [1] stats graphics grDevices utils datasets methods base other attached packages: [1] lme4_0.999375-24 Matrix_0.999375-11 lattice_0.17-8 loaded via a namespace (and not attached): [1] grid_2.7.1 tools_2.7.1 2. Mac OS; R version 2.7.1:
sessionInfo()
R version 2.7.1 (2008-06-23) powerpc-apple-darwin8.10.1 locale: C attached base packages: [1] stats graphics grDevices utils datasets methods base other attached packages: [1] lme4_0.999375-22 Matrix_0.999375-10 lattice_0.17-8 loaded via a namespace (and not attached): [1] grid_2.7.1 3. I also got the segfault on the 64-bit linux machine using R
version
2.6.1, lme4 version 0.99875-9 (I think), and Matrix version 0.999375-9
(I
think). 4. *On the other hand,* it sort of works (without segfaulting at
least)
using older R, lme4 and Matrix on my Mac:
sessionInfo()
R version 2.6.2 (2008-02-08) powerpc-apple-darwin8.10.1 locale: C attached base packages: [1] stats graphics grDevices utils datasets methods base other attached packages: [1] lme4_0.99875-9 Matrix_0.999375-9 lattice_0.17-6 loaded via a namespace (and not attached): [1] grid_2.6.2
mylme <- lmer(myfmla, data=mydata)
Warning messages: 1: In .local(x, ..., value) : Estimated variance-covariance for factor 'by.var' is singular 2: In .local(x, ..., value) : nlminb returned message false convergence (8) --Perhaps the warnings contain a clue as to what the problem is? Thanks for your consideration of this. Ben On 8/13/08 2:55 AM, "Douglas Bates" <bates at stat.wisc.edu> wrote:
I would first try later versions of both the Matrix and lme4
packages.
Versions currently on CRAN are Matrix_0.999375-11 and lme4_0.999375-24 If you continue to get segfaults then having a reproducible example, along with the data - anonymized if you wish, will be important. You can determine the versions of the packages that you are using
with
sessionInfo() On Wed, Aug 13, 2008 at 12:06 AM, Hansen, Ben <bbh at umich.edu> wrote:
Hello all, I am encountering trouble with lme4 on a 64-bit linux system, and
I'm
wondering if any of you might have advice on how to circumvent it.
Using
recent versions of lme4 and R, I was able to compile and install the package, and use it with success on relatively small problems.
However,
when I tried to scale up to work on a big problem, applying lmer()
with
thousands of observations and a hundred or so random effects, my R
fails
reporting a segmentation fault. I don't see the same problem on a different 32-bit Mac OSX system, which runs the same code without evident difficulty. Some details of the setup where the problem occurs. We've tried a
few
combinations of things, so I write what we tried first, with
variants
noted in parentheses:
Red Hat Enterprise Linux 4 (version 5)
R version 2.7.1 (2.6.1)
gcc/g++ compilers for C and C++, Portland Group Fortran compilers
(gcc/g++/gfortran, ie full GNU compiler suite)
Matrix package version 0.999375-10 (installed with and without
--with-package-versions flag)
lme4 version 0.999375-16 (installed with and without
--with-package-versions flag)
Any suggestions? I'd be very appreciative.
Ben
[[alternative HTML version deleted]]
_______________________________________________ R-sig-mixed-models at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models