R on Large Data Sets (again)

Dear R users,

I?ve search the R site for help on this topic but it is hard to find a
precise answer for my questions.

Which are the best options to overcome the RAM memory limitation problems
when using R on ?large? data sets (such as 2 or 3 million records)?

- Is the free available version of R (as opposed to the one provided
by REvolution Computing) compatible with a windows 64-bit machine?
And if I increase the RAM memory enough on win-64, would this
virtually solve my memory limitation problems?

- Is a Unix-like platform a better option than win-64? Again, would
this solve my memory limitation problems?

-----Original Message-----
From: r-help-bounces at r-project.org [mailto:r-help-bounces at r-project.org]
On Behalf Of Jason Morgan
Sent: Saturday, November 28, 2009 6:18 PM
To: Lars Bishop
Cc: r-help at r-project.org
Subject: Re: [R] R on Large Data Sets (again)

Hello Lars,

On 2009.11.28 18:53:09, Lars Bishop wrote:

Dear R users,

I?ve search the R site for help on this topic but it is hard to find a
precise answer for my questions.

Which are the best options to overcome the RAM memory limitation

problems

when using R on ?large? data sets (such as 2 or 3 million records)?

I think you'll have to provide a more precise definition of
"large"---are we talking 1 GB of records or 100 GB? Also, it would help
to know what you are trying to do with the data. The documentation for
the biglm and bigmemory packages may provide some help.

- Is the free available version of R (as opposed to the one provided
by REvolution Computing) compatible with a windows 64-bit machine?
And if I increase the RAM memory enough on win-64, would this
virtually solve my memory limitation problems?

I'm not familiar enough with the commercial version of R, but I do
believe it provides better support for parallelization, which may be of
some help. I don't think, however, that this version will "solve" your
problem.

- Is a Unix-like platform a better option than win-64? Again, would
this solve my memory limitation problems?

Possibly, but Win64 should provide plenty of memory (I believe Windows 7
Ultimate can use up to 192 GB of memory). You just have to find the
system that can take that much... With Unix/Linux you can probably cut
back some overhead, and the memory management is most likely better, but
unless you need to go over 192GB of memory, you don't necessarily have
to move to a different platform.

~Jason

Dear R users,

I?ve search the R site for help on this topic but it is hard to find a
precise answer for my questions.

Which are the best options to overcome the RAM memory limitation problems
when using R on ?large? data sets (such as 2 or 3 million records)?

-          Is the free available version of R (as opposed to the one
provided by REvolution Computing) compatible with a windows 64-bit machine?
And if I increase the RAM memory enough on win-64, would this virtually
solve my memory limitation problems?

-          Is a Unix-like platform a better option than win-64? Again, would
this solve my memory limitation problems?

-          Any better option?
Thanks in advance for your help,
Lars.

	[[alternative HTML version deleted]]



------------------------------------------------------------------------

- Is a Unix-like platform a better option than win-64? Again, would
this solve my memory limitation problems?

Possibly, but Win64 should provide plenty of memory (I believe Windows 7
Ultimate can use up to 192 GB of memory). You just have to find the
system that can take that much... With Unix/Linux you can probably cut
back some overhead, and the memory management is most likely better, but
unless you need to go over 192GB of memory, you don't necessarily have
to move to a different platform.

~Jason

Windows 64-bit can certainly handle large memory spaces, but unless
something has changed recently it my understanding Revolution
Computing's 64-bit is the only 64-bit version of R available for
Windows (due to the unavailability of adequate open source compilers
for 64-bit Windows).  So 64-bit R will need to be Revolution's
solution or a non-Windows platform.

On 2009.11.28 21:50:09, Daniel Nordlund wrote:

- Is a Unix-like platform a better option than win-64? Again, would
this solve my memory limitation problems?

Possibly, but Win64 should provide plenty of memory (I believe Windows 7
Ultimate can use up to 192 GB of memory). You just have to find the
system that can take that much... With Unix/Linux you can probably cut
back some overhead, and the memory management is most likely better, but
unless you need to go over 192GB of memory, you don't necessarily have
to move to a different platform.

~Jason

Windows 64-bit can certainly handle large memory spaces, but unless
something has changed recently it my understanding Revolution
Computing's 64-bit is the only 64-bit version of R available for
Windows (due to the unavailability of adequate open source compilers
for 64-bit Windows).  So 64-bit R will need to be Revolution's
solution or a non-Windows platform.

It appears that GNU does have a project that has had some success at
compiling 64 bit Windows applications:

http://mingw-w64.sourceforge.net/

Not sure if all of the pieces are there for an R build, though.

On 2009.11.28 21:50:09, Daniel Nordlund wrote:

- Is a Unix-like platform a better option than win-64? Again, would
this solve my memory limitation problems?

Possibly, but Win64 should provide plenty of memory (I believe Windows 7
Ultimate can use up to 192 GB of memory). You just have to find the
system that can take that much... With Unix/Linux you can probably cut
back some overhead, and the memory management is most likely better, but
unless you need to go over 192GB of memory, you don't necessarily have
to move to a different platform.

~Jason

Windows 64-bit can certainly handle large memory spaces, but unless
something has changed recently it my understanding Revolution
Computing's 64-bit is the only 64-bit version of R available for
Windows (due to the unavailability of adequate open source compilers
for 64-bit Windows).  So 64-bit R will need to be Revolution's
solution or a non-Windows platform.

It appears that GNU does have a project that has had some success at
compiling 64 bit Windows applications:

http://mingw-w64.sourceforge.net/

Not sure if all of the pieces are there for an R build, though.

-- 
Jason W. Morgan
Graduate Student
Department of Political Science
*The Ohio State University*
154 North Oval Mall
Columbus, Ohio 43210

Windows 64-bit can certainly handle large memory spaces, but unless
something has changed recently it my understanding Revolution
Computing's 64-bit is the only 64-bit version of R available for
Windows (due to the unavailability of adequate open source compilers
for 64-bit Windows).  So 64-bit R will need to be Revolution's
solution or a non-Windows platform.

Or use a commercial Windows compiler.

It appears that GNU does have a project that has had some success at
compiling 64 bit Windows applications:

http://mingw-w64.sourceforge.net/

Well, some interesed people have a project to port GCC and binutils: 
as far as I am aware that is not an official GNU project.

Not sure if all of the pieces are there for an R build, though.

You are welcome to show us how to do it (on the R-devel list): several 
people have spent man months attempting this (including submitting 
many patches to that project), and the rw-FAQ did tell you do so in 
http://cran.r-project.org/bin/windows/base/rw-FAQ.html#How-can-I-compile-R-from-source_003f