allocMatrix error

On Tue, 17 Feb 2009, Hamid Ashafi wrote:

On Sat, Feb 14, 2009 at 00:17,  <ashrafi at ucdavis.edu> wrote:

Hi,

I was trying to read ~400 chips in an affybatch and I got the same message.
Could you find a remedy for that. My server has 128 GB of RAM. However, R
halted ever before it uses the memory.

We don't have anything like sufficient details (please do read the
posting guide).

If the issue is the size of matrices, you possibly (depending on the
compiler) could arrange to compile R (and any relevant system
libraries) to use 64-bit ints.  For C code in R there is typedef to
change, and you would need integer*8 in the Fortran.  We would be
interested to know the results if you do so, but the developers are
unlikely to do so for you.

In any case, since you mention 'affybatch' it looks like this might be
a design issue in that BioC package and the BioC lists might be the
appropriate place to discuss it.  It is not obvious to me why ~400
datasets need a single large R object rather than, say, a list of 400
smaller ones, if that is indeed the problem.  So, to return to my
first point:

We don't have anything like sufficient details.

Please give us the full details of your system, the memory in use (see
?gc) and what you were trying to do.

I have been able to load upto 250 CEL files but this time I wanted to test
what would happen if I want to normalize 400 chips.

R can handle up to 16GB objects, which even for a 64-bit OS and 128GB
of RAM are pretty large objects and do not arise naturally from many
small files.

Thanks for your prompt response.



Hamid

Martin Maechler wrote:

"VK" == Vadim Kutsyy <vadim at kutsyy.com>

    on Fri, 01 Aug 2008 07:35:01 -0700 writes:

    VK> Martin Maechler wrote:

   >>

    VK> The problem is in array.c, where allocMatrix check for

    VK> "if ((double)nrow * (double)ncol > INT_MAX)".  But why

    VK> itn is used and not long int for indexing? (max int is

    VK> 2147483647, max long int is 9223372036854775807)

   >>

   >> Well, Brian gave you all info:

   >>

    VK> exactly, and given that most modern system used for

    VK> computations (i.e.  64bit system) have long int which is

    VK> much larger than int, I am wondering why long int is not

    VK> used for indexing (I don't think that 4 bit vs 8 bit

    VK> storage is an issue).

   >> Did you really carefully read ?Memory-limits ??

   >>

    VK> Yes, it is specify that 4 bit int is used for indexing

    VK> in all version of R, but why? I think 2147483647

    VK> elements for a single vector is OK, but not as total

    VK> number of elements for the matrix.  I am running out of

    VK> indexing at mere 10% memory consumption.

Hmm, do you have 160 GBytes of RAM?

But anyway, let's move this topic from R-help to R-devel.

   [...........]

    VK> PS: I have no problem to go and modify C code, but I am

    VK> just wondering what are the reasons for having such

    VK> limitation.

This limitation and its possible remedies are an interesting topic,

but really not for R-help:

It will be a lot about C programming the internal represenation of R

objects, etc.

Very fascinating .... but for R-devel.

"See you there!"

Martin

R-help at r-project.org mailing list

Quoted from:

http://www.nabble.com/allocMatrix-limits-tp18763791p18776531.html

	[[alternative HTML version deleted]]

-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595