Skip to content

Once again: Error: cannot allocate vector of size

6 messages · Werner W., Steve Lianoglou, werner w +2 more

#
Hi,

I have browsed the help list and looked at the FAQ but I don't find conclusive evidence if this is normal or I am doing something wrong. 
I am running a lm() on a data.frame with 27136 observations of 6 variables (3 num and 3 factor).
After a while R throws this:
Error: cannot allocate vector of size 203.7 MB

This is a Windows XP 32 bit machine with 4 GB in it so that theoretically, R should be able to claim close to 2 GB. 
This is the gc() after the regression:
          used (Mb) gc trigger  (Mb)  max used   (Mb)
Ncells  272299  7.3     875833  23.4   1368491   36.6
Vcells 4526037 34.6  116536251 889.2 145524997 1110.3
[1] 1230.25
[1] 47.89
Looking at memory.size, R should be easily able to allocate that space, shouldn't it?

Many thanks for any hints!

--Werner

__________________________________________________
Do You Yahoo!?
Sie si
n Massenmails. 
http://mail.yahoo.com
#
Please re-read the posting guide e.g. you didn't provide an example data set 
or a way to generate one,  or any R version information.

"Werner W." <pensterfuzzer at yahoo.de> wrote in message 
news:646146.32238.qm at web23002.mail.ird.yahoo.com...
#
Hi Werner,
On Thu, Jan 21, 2010 at 9:23 PM, Werner W. <pensterfuzzer at yahoo.de> wrote:
AFAIK, the error message "Error: cannot allocate vector of size 203.7
MB" doesn't mean that your entire data set is 203.7MB and it can't
load it ... but rather R just tried grab another chunk of RAM that
size, and the OS has denied it that favor ... so, I'm guessing it's
getting "half way" through loading/running on your data, but can't
complete the task because it has already chewed through all the RAM
that at has access to.

You can look at your RAM usage (through the task manager? sorry,
haven't used windows in a long time) while R is doing its thing to
verify until a windows guru chimes in.

-steve
#
Thanks Matthew, you are absolutely right.

I am working on Windows XP SP2 32bit with R versions 2.9.1.

Here is an example:
  d <- as.data.frame(matrix(trunc(rnorm(6*27136, 10000, 100)),ncol=6))
  d[,4:5] <- trunc(100*runif(2*27136, 0, 1))
  d[,6] <- trunc(1000*runif(27136, 0, 1))
  for (i in 4:6) d[,i] <- as.factor(d[,i])
  lm(V1 ~ log(V2) + log(V3) + V4 + V5 + V6, data=d)
  memory.size(max=F)
  memory.size(max=T)

I managed to get it run through after setting the 3GB switch for Windows and
with a clean R session.
I also noticed later, that after removing na.action=na.exclude more
regressions run through.

But before and after the lm() it seems there should be enough memory which
means that lm() builds up some quite large objects during its computations?
#
Fantastic. You're much more likely to get a response now.  Best of luck.

"werner w" <pensterfuzzer at yahoo.de> wrote in message 
news:1264175935970-1100164.post at n4.nabble.com...
#
The problem I see here is not so much the 27136 observations, but the fact that the first two factors have up to 101 different levels and the third factor up to 1001 different levels. That means that lm() is essentially creating 100+100+1000 dummies for those factors, leading to a large (and sparse) design matrix (containing over 30 million entries).

Best,

--
Wolfgang Viechtbauer                        http://www.wvbauer.com/
Department of Methodology and Statistics    Tel: +31 (43) 388-2277
School for Public Health and Primary Care   Office Location:
Maastricht University, P.O. Box 616         Room B2.01 (second floor)
6200 MD Maastricht, The Netherlands         Debyeplein 1 (Randwyck)


----Original Message----
From: r-help-bounces at r-project.org
[mailto:r-help-bounces at r-project.org] On Behalf Of Matthew Dowle Sent:
Friday, January 22, 2010 17:33 To: r-help at stat.math.ethz.ch
Subject: Re: [R] Once again: Error: cannot allocate vector of size