R on Large Data Sets (again)

Sat, Nov 28, 2009 9:50 PM

-----Original Message-----
From: r-help-bounces at r-project.org [mailto:r-help-bounces at r-project.org]
On Behalf Of Jason Morgan
Sent: Saturday, November 28, 2009 6:18 PM
To: Lars Bishop
Cc: r-help at r-project.org
Subject: Re: [R] R on Large Data Sets (again)

Hello Lars,

On 2009.11.28 18:53:09, Lars Bishop wrote:

Dear R users,

I?ve search the R site for help on this topic but it is hard to find a
precise answer for my questions.

Which are the best options to overcome the RAM memory limitation

problems

when using R on ?large? data sets (such as 2 or 3 million records)?

I think you'll have to provide a more precise definition of
"large"---are we talking 1 GB of records or 100 GB? Also, it would help
to know what you are trying to do with the data. The documentation for
the biglm and bigmemory packages may provide some help.

- Is the free available version of R (as opposed to the one provided
by REvolution Computing) compatible with a windows 64-bit machine?
And if I increase the RAM memory enough on win-64, would this
virtually solve my memory limitation problems?

I'm not familiar enough with the commercial version of R, but I do
believe it provides better support for parallelization, which may be of
some help. I don't think, however, that this version will "solve" your
problem.

- Is a Unix-like platform a better option than win-64? Again, would
this solve my memory limitation problems?

Possibly, but Win64 should provide plenty of memory (I believe Windows 7
Ultimate can use up to 192 GB of memory). You just have to find the
system that can take that much... With Unix/Linux you can probably cut
back some overhead, and the memory management is most likely better, but
unless you need to go over 192GB of memory, you don't necessarily have
to move to a different platform.

~Jason

Windows 64-bit can certainly handle large memory spaces, but unless something has changed recently it my understanding Revolution Computing's 64-bit is the only 64-bit version of R available for Windows (due to the unavailability of adequate open source compilers for 64-bit Windows).  So 64-bit R will need to be Revolution's solution or a non-Windows platform.

Hope this is helpful,

Dan 

Daniel Nordlund
Bothell, WA USA

R on Large Data Sets (again)

Thread (8 messages)