An embedded and charset-unspecified text was scrubbed... Name: not available URL: <https://stat.ethz.ch/pipermail/r-help/attachments/20120206/5bf6dda4/attachment.pl>
R's memory capabilities
6 messages · Alaios, Peter Dalgaard, Ben Bolker +2 more
On Feb 6, 2012, at 11:15 , Alaios wrote:
Dear all, I have tried to create a diagonal matrix of size diag(65536) I am getting the message that the array function can not create so big array as the value I ask is larger than the <environment: namespace:base>
.Machine$integer
[1] 2147483647 as you will see my .Machine$integer is the one above. What that variable means? I am using R in a system that has 500 GB of ram available. Would it be something wrong in the installation?
Well, it means what it says. Indexing is limited by the size of a 32-bit integer and (2^16)^2 is too big for that. So you are limited to roughly 16 GB for any single object in R. This issue has been foreseeable for some time. It can only be fixed by internal changes in the R engine, but switching to long integers has various issues, so it is not a straightforward modification. Meanwhile, you might want to consider whether you really do need a diagonal matrix of that size stored in full. It's an awful lot of zeroes to carry around....
Regards Alex [[alternative HTML version deleted]]
______________________________________________ R-help at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Peter Dalgaard, Professor Center for Statistics, Copenhagen Business School Solbjerg Plads 3, 2000 Frederiksberg, Denmark Phone: (+45)38153501 Email: pd.mes at cbs.dk Priv: PDalgd at gmail.com
An embedded and charset-unspecified text was scrubbed... Name: not available URL: <https://stat.ethz.ch/pipermail/r-help/attachments/20120206/bf67ff42/attachment.pl>
peter dalgaard <pdalgd <at> gmail.com> writes:
On Feb 6, 2012, at 11:15 , Alaios wrote:
Dear all, I have tried to create a diagonal matrix of size diag(65536)
[snip ...]
Well, it means what it says. Indexing is limited by the size of a 32-bit integer and (2^16)^2 is too big for that. So you are limited to roughly 16 GB for any single object in R. This issue has been foreseeable for some time. It can only be fixed by internal changes in the R engine, but switching to long integers has various issues, so it is not a straightforward modification. Meanwhile, you might want to consider whether you really do need a diagonal matrix of that size stored in full. It's an awful lot of zeroes to carry around....
What Peter is hinting at (I think) is that you depending
on your application you may be able to get away with using
sparse matrices, which only store the positions and values
of non-zero elements rather than the values of all elements.
For example:
library(Matrix)
d <- Diagonal(2^16)
str(d)
Formal class 'ddiMatrix' [package "Matrix"] with 4 slots
..@ diag : chr "U"
..@ Dim : int [1:2] 65536 65536
..@ Dimnames:List of 2
.. ..$ : NULL
.. ..$ : NULL
..@ x : num(0)
object.size(d)
652 bytes
vignette("Intro2Matrix")
-----Original Message----- From: r-help-bounces at r-project.org [mailto:r-help-bounces at r-project.org] On Behalf Of Alaios Sent: Monday, February 06, 2012 3:11 AM To: peter dalgaard Cc: R-help at r-project.org Subject: Re: [R] R's memory capabilities Thanks a lot for your answer :) actually I am converting a 256*256 image to a vector so to apply some 1-d transformations. Is it possible to create in R a sparse matrix instead of carrying all those zeros? So that means that I want to have a diagonal sparse matrix.. Thanks a lot Alex
maybe look at ?as.vector. But it is really hard to know without knowing what you expect your "vector to look like or what transforms you will be doing. Hope this is helpful, Dan Daniel Nordlund Bothell, WA USA
Use the Matrix package which provides easy access to sparse matrices. I think (unchecked) there's even easy provision for a diagonal sparse matrix. That said, if you are always working with diagonal matrices, couldn't you just keep the diagonals as vectors since the formulas for matrix multiplication reduce nicely in that case? Michael
On Mon, Feb 6, 2012 at 6:10 AM, Alaios <alaios at yahoo.com> wrote:
Thanks a lot for your answer :) actually I am converting a 256*256 image to a vector so to apply some 1-d transformations. Is it possible to? create in R a sparse matrix instead of carrying all those zeros? So that means that I want to have a diagonal sparse matrix.. Thanks a lot Alex
________________________________ ?From: peter dalgaard <pdalgd at gmail.com> Cc: "R-help at r-project.org" <R-help at r-project.org> Sent: Monday, February 6, 2012 11:54 AM Subject: Re: [R] R's memory capabilities On Feb 6, 2012, at 11:15 , Alaios wrote: Dear all, I have tried to create a diagonal matrix of size diag(65536) I am getting the message that the array function can not create so big array as the value I ask is larger than the <environment: namespace:base> .Machine$integer [1] 2147483647 as you will see my .Machine$integer is the one above. What that variable means? I am using R in a system that has 500 GB of ram available. Would it be something wrong in the installation? Well, it means what it says. Indexing is limited by the size of a 32-bit integer and (2^16)^2 is too big for that. So you are limited to roughly 16 GB for any single object in R. This issue has been foreseeable for some time. It can only be fixed by internal changes in the R engine, but switching to long integers has various issues, so it is not a straightforward modification. Meanwhile, you might want to consider whether you really do need a diagonal matrix of that size stored in full. It's an awful lot of zeroes to carry around.... Regards Alex ??? [[alternative HTML version deleted]] ______________________________________________ R-help at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. -- Peter Dalgaard, Professor Center for Statistics, Copenhagen Business School Solbjerg Plads 3, 2000 Frederiksberg, Denmark Phone: (+45)38153501 Email: pd.mes at cbs.dk? Priv: PDalgd at gmail.com ? ? ? ?[[alternative HTML version deleted]] ______________________________________________ R-help at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.