Memory limit for Windows 64bit build of R

An embedded and charset-unspecified text was scrubbed...
Name: not available
URL: <https://stat.ethz.ch/pipermail/r-help/attachments/20120806/761a955f/attachment.pl>
An embedded and charset-unspecified text was scrubbed...
Name: not available
URL: <https://stat.ethz.ch/pipermail/r-help/attachments/20120805/9b87e89f/attachment.pl>

Dear all

I have a Windows Server 2008 R2 Enterprise machine, with 64bit R  
installed
running on 2 x Quad-core Intel Xeon 5500 processor with 24GB DDR3  
1066 Mhz
RAM.  I am seeking to analyse very large data sets (perhaps as much as
10GB), without the addtional coding overhead of a package such as
bigmemory().
It may depend in part on how that number is arrived at. And what you  
plan on doing with it. (Don't consider creating a dist-object.)
My question is this - if we were to increase the RAM on the machine to
(say) 128GB, would this become a possibility?  I have read the
documentation on memory limits and it seems so, but would like some
additional confirmation before investing in any extra RAM.
The trypical advices is you will need memory that is 3 times as large  
as a large dataset, and I find that even more headroom is needed. I  
have 32GB and my larger datasets occupy 5-6 GB and I generally have  
few problems. I had quite a few problems with 18 GB, so I think the  
ratio should be 4-5 x your 10GB object.  I predict you could get by  
with 64GB. (please send check for half the difference in cost between  
64GB abd 128 GB.)
David.

> Kind regards
>
> Alan
>
> Alan Simpson
> Technical Lead, Retail Model Development
> Retail Models Project
> National Australia Bank
>
> Level 15, 500 Bourke St, Melbourne VIC
> Tel: +61 (0) 3 8697 7135  |  Mob: +61 (0) 412 975 955
> Email: Alan.X.Simpson at nab.com.au
>
>
> The information contained in this email and its attachments may be  
> confidential.
> If you have received this email in error, please notify the sender  
> by return email,
> delete this email and destroy any copy.
>
> Any advice contained in this email has been prepared without taking  
> into
> account your objectives, financial situation or needs. Before acting  
> on any
> advice in this email, National Australia Bank Limited ABN 12 004 044  
> 937 AFSL and Australian Credit Licence 230686 (NAB) recommends that
> you consider whether it is appropriate for your circumstances.
> If this email contains reference to any financial products, NAB  
> recommends
> you consider the Product Disclosure Statement (PDS) or other  
> disclosure
> document available from NAB, before making any decisions regarding any
> products.
>
> If this email contains any promotional content that you do not wish  
> to receive,
> please reply to the original sender and write "Don't email promotional
> material" in the subject.
>
>
> 	[[alternative HTML version deleted]]
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

David Winsemius, MD
Alameda, CA, USA
On Aug 5, 2012, at 3:52 PM, Alan.X.Simpson at nab.com.au wrote:

Dear all

I have a Windows Server 2008 R2 Enterprise machine, with 64bit R
installed
running on 2 x Quad-core Intel Xeon 5500 processor with 24GB DDR3 1066
Mhz
RAM.  I am seeking to analyse very large data sets (perhaps as much as
10GB), without the addtional coding overhead of a package such as
bigmemory().
It may depend in part on how that number is arrived at. And what you
plan on doing with it. (Don't consider creating a dist-object.)
My question is this - if we were to increase the RAM on the machine to
(say) 128GB, would this become a possibility?  I have read the
documentation on memory limits and it seems so, but would like some
additional confirmation before investing in any extra RAM.
The trypical advices is you will need memory that is 3 times as large as
a large dataset, and I find that even more headroom is needed. I have
32GB and my larger datasets occupy 5-6 GB and I generally have few
problems. I had quite a few problems with 18 GB, so I think the ratio
should be 4-5 x your 10GB object.  I predict you could get by with 64GB.
(please send check for half the difference in cost between 64GB abd 128
GB.)

10Gb objects should be fine, but note that a vector/array/matrix cannot 
exceed  2^31-1 elements, hence a 17Gb vector/matrix/array of doubles / 
reals.

Best,
Uwe Ligges

On 06.08.2012 09:34, David Winsemius wrote:
On Aug 5, 2012, at 3:52 PM, Alan.X.Simpson at nab.com.au wrote:

Dear all

I have a Windows Server 2008 R2 Enterprise machine, with 64bit R
installed
running on 2 x Quad-core Intel Xeon 5500 processor with 24GB DDR3 1066
Mhz
RAM.  I am seeking to analyse very large data sets (perhaps as much as
10GB), without the addtional coding overhead of a package such as
bigmemory().
It may depend in part on how that number is arrived at. And what you
plan on doing with it. (Don't consider creating a dist-object.)
My question is this - if we were to increase the RAM on the machine to
(say) 128GB, would this become a possibility?  I have read the
documentation on memory limits and it seems so, but would like some
additional confirmation before investing in any extra RAM.
The trypical advices is you will need memory that is 3 times as large as
a large dataset, and I find that even more headroom is needed. I have
The advice is 'at least 3 times'.  It all depends what you are doing 
(and how slow your swap is -- on Windows it is likely to be slow; on a 
Linux box with a fast SSD it can be viable to use swap).
32GB and my larger datasets occupy 5-6 GB and I generally have few
problems. I had quite a few problems with 18 GB, so I think the ratio
should be 4-5 x your 10GB object.  I predict you could get by with 64GB.
But 3 x 18GB > 32GB!
(please send check for half the difference in cost between 64GB abd 128
GB.)

10Gb objects should be fine, but note that a vector/array/matrix cannot
exceed  2^31-1 elements, hence a 17Gb vector/matrix/array of doubles /
reals.
That's true for R 2.15.1, but not the development version.  Further, 
R-devel makes substantially fewer copies of objects, most of which 
improvements have been ported to R-patched.

dist() is one example of substantial improvements.
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595