RandomForest, Party and Memory Management

Hi Lorenzo:

On Sun, Feb 3, 2013 at 11:47 AM, Lorenzo Isella
<lorenzo.isella at gmail.com> wrote:

Dear All,
For a data mining project, I am relying heavily on the RandomForest and
Party packages.
Due to the large size of the data set, I have often memory problems (in
particular with the Party package; RandomForest seems to use less memory). I
really have two questions at this point
1) Please see how I am using the Party and RandomForest packages. Any
comment is welcome and useful.

As noted elsewhere, the example is not reproducible so I can't help you there.

myparty <- cforest(SalePrice ~ ModelID+
                   ProductGroup+
                   ProductGroupDesc+MfgYear+saledate3+saleday+
                   salemonth,
                   data = trainRF,
control = cforest_unbiased(mtry = 3, ntree=300, trace=TRUE))




rf_model <- randomForest(SalePrice ~ ModelID+
                    ProductGroup+
                    ProductGroupDesc+MfgYear+saledate3+saleday+
                    salemonth,
                    data = trainRF,na.action = na.omit,
   importance=TRUE, do.trace=100, mtry=3,ntree=300)

2) I have another question: sometimes R crashes after telling me that it is
unable to allocate e.g. an array of 1.5 Gb.
However, I have 4Gb of ram on my box, so...technically the memory is there,
but is there a way to enable R to use more of it?

4Gb is not a lot of RAM for data mining projects. I have twice that
and run into memory limits on some fairly simple tasks (e.g., 2D
tables) in large simulations with 1M or 10M runs. Part of the problem
is that data is often copied, sometimes more than once. If you have a
1Gb input data frame, three copies and you're out of space. Moreover,
copied objects need contiguous memory, and this becomes very difficult
to achieve with large objects and limited RAM. With 4Gb RAM, you need
to be more clever:

* eliminate as many other processes that access RAM as possible (e.g.,
no active browser)
* think of ways to process your data in chunks (which is harder to do
when the objective is model fitting)
* type ?"Memory-limits"  (including the quotes) at the console for
explanations about memory limits and a few places to look for
potential solutions
* look into 'big data' packages like ff or bigmemory, among others
* if you're in an (American ?) academic institution, you can get a
free license for Revolution R, which is supposed to be better for big
data problems than vanilla R

It's hard to be specific about potential solutions, but the above
should broaden your perspective on the big data problem and possible
avenues for solving it.

Dennis

Many thanks

Lorenzo