Datum: Wed, 21 Dec 2011 15:47:36 -0600
Von: Douglas Bates <bates at stat.wisc.edu>
An: cumuluss at web.de
CC: r-sig-mixed-models at r-project.org, "a.r.runnalls" <a.r.runnalls at kent.ac.uk>
Betreff: Re: [R-sig-ME] problems with allocate memory
On Tue, Dec 20, 2011 at 5:25 PM,? <cumuluss at web.de> wrote:
Hi Douglas,
The variable?d? has about 710 levels.
For your other request I tried to fit the suggested model but it was not
possible. I tried it with different approaches, first without any
interactions and non nonlinear term. It fitted. The object size was about 731571664
bytes. Then I successive made the model more complex. With one two way
interaction, with one three way interaction or with the nonlinear term it was
slightly the same as before. With five two way interaction always with the
nonlinear term the object size went up to 1075643424 bytes. With one
additional two way interaction the model won?t fit anymore with the known
error.
Which is an indication that the fixed-effects model matrix is getting
to be too large.? There are no simple solutions at present.? You may
find that some packages allow you to fit such large models by working
with horizonal chunks of the data and accumulating the result but
extending those to GLMMs would be decidedly non-trivial.
Perhaps another hint: Yesterday I attempted to fit a much simpler model
with lmer, just to see if this works. (mfit=lmer(gr.b ~ f.ag + f.se + o.se
+ diff + exp.r + kl + (1|f)+(1|o), data=c.data, family=binomial)). It
fitted but I could not open mfit. Trying to see only the coefficients also did
not work. I saved the image and this one is unfamiliar huge about 1.7 GB.
The problem there is that the implicit print(mfit) (which is what I
imagine you mean when you say "could not open") ends up taking copies
of the whole object, which will eat up all your memory.? Development
versions of lme4 may eventually help with that.
By the way: After reloading the image some interesting things happened.
An error occurred: slot coefs are not an S4 object. It seems to me that it
is not possible to save the model results in an R image. Is that right?
It should be possible to save and load such an object but there is
always the problem that when you have an object that is a sizable
fraction of the total available memory then you can get bitten if
something behind the scenes happens to take a copy at some point in
the calculation.? The original design in R for keeping track of when a
copy must be made is not the greatest and, as a result, R is somewhat
conservative when deciding whether or not to copy an object.? Getting
around that limitation would mean reimplementing R, more-or-less from
scratch and Andrew Runnalls is the only person I know who is willing
to embark on that.
-------- Original-Nachricht --------
Datum: Tue, 20 Dec 2011 09:23:37 -0600
Von: Douglas Bates <bates at stat.wisc.edu>
An: cumuluss at web.de
CC: r-sig-mixed-models at r-project.org
Betreff: Re: [R-sig-ME] problems with allocate memory
On Mon, Dec 19, 2011 at 5:54 PM,? <cumuluss at web.de> wrote:
Hi Douglas,
thanky you for your reply. This is "mydata"
'data.frame':?? 3909896 obs. of? 19 variables:
? $ gr.b??????????? : int? 0 0 0 0 0 0 0 0 0 0 ...
? $ o.ag????? : num? -0.651 -0.651 -0.651 -0.651 -0.651 ...
? $ o.rar????? : num? -0.935 -0.935 -0.935 -0.935 -0.935 ...
? $ si?????? : num? 0.299 0.299 0.299 0.299 0.299 ...
? $ f.ag?????? : num? -1.25 -1.36 -1.33 -1.26 -1.21 ...
? $ f.se?????? : Factor w/ 2 levels "F","M": 1 2 1 2 2 2 1 1 1
? $ o.se?????? : Factor w/ 2 levels "F","M": 1 1 1 1 1 1 1 1 1
? $ diff??????? : num? -0.536 -0.514 -0.521 -0.534 -0.545 ...
? $ exp.r?????????? : num? -0.168 -0.168 -0.163 -0.168
? $ f.rar????? : num? -0.911 0.215 1.224 -1.086 1.107 ...
? $ f.si: num? 1.0008 1.1583 0.0561 -0.4163 0.371 ...
? $ kl????????????? : Factor w/ 3 levels
? $ sn????????? : Factor w/ 2 levels "BS","MS": 1 1 1 1 1 1
? $ MP_y_n????????? : Factor w/ 2 levels "0","1": 2 2 2 2 2
? $ ratio?????? : num? -0.0506 -0.0506 -0.0506 -0.0506 -0.0506
? $ f?????????? : Factor w/ 55 levels "0A0","0A1","0A2",..:
6 7 8 9 10 11 13 15 16 ...
? $ o?????????? : Factor w/ 552 levels "","00T","00Z",..: 2
? $ d??????????? : int? 9099 9099 9099 9099 9099 9099
? $ MP????????????? : num? 6 6 6 6 5 4 6 6 6 6 ...
formula for the model:
mfit=lmer(gr.b ~ o.ag + o.rar + si + ((f.ag + I(f.ag^2)) * (f.se *
+ diff + exp.r + f.rar + f.si + kl + sn + MP_y_n + ratio))) +
(1|f)+(1|o)+(1|d) + offset(log(MP)), data=c.data, family=binomial)
I hope this is what you want to see. Thank you for your help.
My guess is that the problem is with creating the fixed-effects model
matrix, of which there could be several copies created during the
evaluation and optimization of the deviance.
Just as a test, could you fit the model for the fixed-effects only
using glm and check on what the size of the model matrix is?
Something like
glm1 <- glm(gr.b ~ o.ag + o.rar + si + ((f.ag + I(f.ag^2)) * (f.se *
(o.se + diff + exp.r + f.rar + f.si + kl + sn + MP_y_n + ratio))) +
offset(log(MP)), data=c.data, family=binomial)
object.size(model.matrix(glm1)
Also, could you convert 'd' to a factor and run str again so we can
learn how many levels there are?? Either that or send the result of
length(unique(mydata$d))
-------- Original-Nachricht --------
Datum: Mon, 19 Dec 2011 14:50:06 -0600
Von: Douglas Bates <bates at stat.wisc.edu>
An: cumuluss at web.de
CC: r-sig-mixed-models at r-project.org
Betreff: Re: [R-sig-ME] problems with allocate memory
On Sun, Dec 18, 2011 at 3:17 PM,? <cumuluss at web.de> wrote:
I have been trying to fit a glmm with a binomial error structure.
model is a little bit complex. I have 8 continuous predictor
them as nonlinear term, 5 categorical predictor variables with some
three-way interactions between them. Additional I have 3 random
offset variable in the model. Number of obs is greater than
I?m working with the latest version of R 2.14.0 on a 64 bit
Everything I tried (reducing model complexity, different 64bit PC
even more memory) nothing leads to a fitted model, always the Error
cannot allocate vector of size 2GB.
Is there anything I can do? I would be very grateful for any
You probably have multiple copies of some large objects hanging
around.? Can you send us the output of
str(myData)
where 'myData' is the name of the model frame containing the data
are using and the formula for the model you are trying to fit?