Crossed random effects
Thanks for the clarification. It is no secret that large plant-breeding programs (both corporate and governmental--see http://www.dpi.nsw.gov.au/__data/assets/pdf_file/113474/annual_report_part_3.pdf) have adopted ASREML, probably due to the "war stories" with crossed random effects that you mention. I have heard several people say that ASREML is often orders of magnitude (100-1000) times better than SAS for handling large datasets with crossed random effects. My limited experience suggests ASREML/Genstat/SAMM and lme4 are in the same order-of-magnitude performance-wise. P.S. I offer sincere appreciation for the "Mixed-effects modeling with crossed random effects for subjects and items" paper, particularly the MCMC approach and the corresponding interpretations and discussions. Very nice. K Wright
On 3/13/07, Douglas Bates <bates at stat.wisc.edu> wrote:
On 3/13/07, Kevin Wright <kw.statr at gmail.com> wrote:
I am confused by some apparent contradictions about fitting crossed random effects in software. Consider this quote from http://www.mpi.nl/world/persons/private/baayen/publications/baayenDavidsonBates.pdf "To our knowledge, the only software currently available for fitting mixed-effects models with crossed random effects is the lme4 package"
That statement should have been more carefully worded. It is in reference to the types of experimental situations described in that paper where random effects are associated with subject and item, subjects are crossed with item and the numbers of both the subjects and the items can be very large.
Yet, nlme and GLIMMIX appear to claim that crossed-random effects can be fit by those respective tools: In Mixed Effects Models in S and S-Plus: "The crossed random-effects structure is represented in lme by a combination of pdBlocke3d and pdIdent objects" (page 163)
It is possible to fit a model with crossed random effects with lme provided that the number of levels of both of the crossed factors is small. Otherwise you end up with huge, sparse model matrices that are being treated as dense matrices and you quickly run out of memory or time or both. Really, doesn't a random effects specification like pdBlocked(list(pdIdent(~ rows - 1), pdIdent(~ columns - 1))) smell like a kludge to you?
http://support.sas.com/rnd/app/papers/glimmix.pdf "The GLIMMIX procedure, on the other hand, determines by default the marginal log likelihood as that of an approximate linear mixed model. This allows multiple random effects, nested and crossed random effects, multiple cluster types, and R-side random components." [and] "Example 2. Mating Experiment with Crossed Random Effects"
I think that several readers of this list could tell you war stories of trying to fit models with crossed random effects using SAS PROC MIXED or SAS PROC NLMIXED versus fitting the same model in lmer or lmer2. You are correct that one can specify a model with crossed random effects in SAS PROC MIXED and that we overstated the uniqueness of the capabilities of lmer to fit such models. However, if you want to try to fit such a model in SAS PROC MIXED when you have large numbers of subjects and large numbers of items you had better be prepared to wait for a long time.
Are these three quotes using different definitions of "crossed random effects"? Have I taken the quotes out of context? Any clarifications would be appreciated. Thanks, K Wright
_______________________________________________ R-sig-mixed-models at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models