An embedded and charset-unspecified text was scrubbed... Name: not available URL: <https://stat.ethz.ch/pipermail/r-help/attachments/20120910/d7f883f1/attachment.pl>
Memory errors using lmer
2 messages · McCall, Ken (CMG-Dayton), Ben Bolker
McCall, Ken (CMG-Dayton <Ken.McCall <at> coxinc.com> writes:
I'm trying to run a linear mixed effects analysis on fairly large datasets with lmer (from the lme4 package) on a 32-bit Windows machine running XP with 3 GB of RAM. It's not working. (details below)
I've researched the ff and bigmemory packages, but it appears they won't handle the mixed mode dataset I'm analyzing. It has some character fields for the categorical variables. It's also not clear a linear mixed effect regression can be run with those packages. Can anyone point me to a lme solution on larger datasets that can address the 32-bit memory limitations?
The smallest dataframe I'm trying to run is about 120,000 observations and 7 variables, but I'd rather run a for loop script on 620K+ observations. I've seen several recommendations (Kabacoff in 'R in Action' and others) that when possible, run R in a 64-bit build. Problem is I'm on a deadline, and procuring a new computer takes time, and approvals up the food chain. Suggestions? Is 64-bit my only option?
install.packages("lme4")
(only needs to be done once)
library(lme4) math07g4 <- sqlQuery(conn, "select ssid, ss_chg,
campus2, district_id, pblack, pfreelnch, pmob
FROM codemob0607ma WHERE grade2 = 4")
Why are you attach()ing? Probably unnecessary ...
attach(math07g4)
fit07ma4 <- lmer(ss_chg ~ 1 + factor(campus2) + factor(district_id) +
pblack + pfreelnch + pmob + (1 | campus2) + (1 | district_id), data=math07g4) And I get this:
Error: cannot allocate vector of size 2.5 Gb In addition: Warning messages: 1: In model.matrix.default(mt, mf, contrasts) : Reached total allocation of 2187Mb: see help(memory.size)
Once upon a time there may have been an option for sparse model matrices, but not now (I think). Depending on whether you have any budget at all, I wonder if you could use Amazon ... google "r amazon ec2 instance" for more information ... If you need more info, I would suggest posting to r-sig-mixed-models <at> r-project.org (a specialty mailing list for mixed models).