Model specification help
On 3/12/07, Andrew Perrin <clists at perrin.socsci.unc.edu> wrote:
Update: I tried the same model on a 10-percent random sample and it completed fine:
grades.10pct.lmer<-lmer(grade.pt ~ (1|stud.id) + (1|instr.id) +
(1|cour.dep),n ewgrades.10pct,control=list(gradient = FALSE, niterEM = 0, msVerbose = 1)) 0 368526.: 0.114490 0.0870205 0.00154286 1 353650.: 0.125362 0.292639 0.980115 2 351398.: 0.222252 0.783158 0.978312 3 351303.: 0.186460 0.779040 0.977594 4 351290.: 0.151424 0.770734 0.976153 5 351274.: 0.168863 0.739213 0.975204 6 351273.: 0.161381 0.726354 0.973485 7 351273.: 0.166878 0.716955 0.963202 8 351269.: 0.167807 0.742046 0.904732 9 351269.: 0.162145 0.739476 0.903375 10 351268.: 0.165429 0.737807 0.898187 11 351238.: 0.165235 0.751137 0.505946 12 351233.: 0.158653 0.728281 0.461245 13 351223.: 0.162827 0.729269 0.360067 14 351221.: 0.165015 0.728947 0.324127 15 351219.: 0.158737 0.728858 0.252385 16 351218.: 0.169401 0.730054 0.229288 17 351217.: 0.164054 0.726687 0.217478 18 351217.: 0.163587 0.723005 0.242674 19 351217.: 0.164458 0.725164 0.235951 20 351217.: 0.164303 0.725227 0.234643 21 351217.: 0.164319 0.725163 0.234738
summary(grades.10pct.lmer)
Linear mixed-effects model fit by REML
Formula: grade.pt ~ (1 | stud.id) + (1 | instr.id) + (1 | cour.dep)
Data: newgrades.10pct
AIC BIC logLik MLdeviance REMLdeviance
351225 351265 -175608 351212 351217
Random effects:
Groups Name Variance Std.Dev.
instr.id (Intercept) 0.067380 0.25958
stud.id (Intercept) 0.297357 0.54530
cour.dep (Intercept) 0.096256 0.31025
Residual 0.410055 0.64036
number of obs: 167654, groups: instr.id, 7198; stud.id, 5471; cour.dep, 97
Fixed effects:
Estimate Std. Error t value
(Intercept) 3.19796 0.03476 92.01
The fact that this fit converged in 21 iterations and the earlier fits to the full data set ran out of memory after 20 iterations makes me suspect that the memory problem is not in the convergence but in the construction of the object to be returned. Right now the only way to cut down on the size of that object is to use the optional argument frame = FALSE in the call to lmer or lmer2. If you have a chance to do a traceback on a fit that runs out of memory, it would be very valuable to know if the insufficient memory condition occurred within the call to nlminb (lmer or old version of lmer2) or .Call(mer2_optimize, mer, cv$msVerbose) (new version of lmer2) versus within a call to new. I suspect it is the latter.