lme4: apparently different results between 0.8-2 and 0.95-6
Steve Cumming wrote:
I've been using lme4 to fit Poisson GLMMs with crossed random effects. The
data are counts(y) sampled at 55 sites over 4 (n=12) or 5 (n=43) years. Most
models use three fixed effects: x1 is a two level factor; x2 and x3 are
continuous. We are including random intercepts for YEAR and SITE. On
subject-matter considerations, we are also including a random coefficient
for x3 within YEAR.
Neglecting the log link, the model is
y_{i,j} = x'_i \beta + \eta_i + z'_i \phi_j + \epsilon_{i,j}
where
i indexes SITE and j indexes YEAR,
\beta is the vector of fixed effects
\eta_i in the random intercept for SITE
and
\phi_j are the random intercept and coefficient for YEAR.
I have written x'_i because the covariates are assumed (reasonably) to be
constant over the 5 years. Thus, obviously, the z'_i = (1, x3_i) are
constant over j as well.
Using lme4 0.8-2 and R 1.9.0 (under Windows), the call
GLMM(y~x1 + x2 + x3,random = list(YEAR=~1+x3, SITE=~1), data=foo,
family=poisson, offset=log(reps))
seemed to work correctly, so far as I can tell. The fixed effects were
more-or-less consistent with those estimated by an ordinary GLM, and the
random YEAR effects had signs, magnitudes and correlation appeared to be
sensible and consistent with my expectations.
Earlier today, we updated to lme4 0.95-6 and R 2.1.0. When we try to use
lmer to fit the same model, it complains bitterly:
lmer(y ~ x1 + x2 + x3 + (1 + x3 | YEAR) + (1 | SITE), data=foo,
family=poisson, offset=log(reps))
Error: Unable to invert singular factor of downdated X'X
Simpler models still work (or at least return):
lmer(y ~ x1 + x2 + x3 + (1 + x3 | YEAR), ...)
lmer(y ~ x1 + x2 + x3 + (1 | YEAR) + (1 | SITE), ...)
As I mentioned, the design is unbalanced. But, we get same "invert singular
factor" Error using the balanced subset.
Can anybody advise? Are we using lmer incorrectly? Or is the new error
perhaps telling us that GLMM in 0.8-2 wasn't actually working in some sense?
I think you are using lmer correctly and that deep in the code there is a glitch related to exactly the circumstances you outlined - a model matrix with more than one column for one grouping factor (i.e. (x3|YEAR)) and a model matrix with a different number of columns for another grouping factor (1|SITE)). Because, as you mentioned, YEAR and SITE are crossed, the internal representation gets more complicated than it would be if you had nested grouping factors. The fact that you can fit the simpler models (thanks for checking that) is a strong indication where the problem is. Is it possible for you to share the data with Deepayan Sarkar and me? If so it would help us track down the bug. Please contact us off list if you can do so. In the meantime I suggest using GLMM for this particular problem if you still have access to the old copy of the lme4 package. If not, contact us off list and we will provide you with a replacement package that can be used in conjunction with the current lme4. This is an exemplary bug report. You did a great job of isolating the problem.