Memory allocation fails in R 2.2.1 and R 2.3.0 on SGI Irix, while plenty of memory available (PR#8861)

Dear R developers,

We have a big SGI Origin computation server with 32 cpu's and 64 Gb of
RAM. In R 2.0.0 we could run large jobs, allocating 8 Gb of RAM was not
a problem, for example by running:

 > v1 <- seq(1,2^29)
 > v2 <- seq(1,2^29)
 > v3 <- seq(1,2^29)
 > v4 <- seq(1,2^29)

This yields an R process, consuming about 8 Gb of RAM:
      PID       PGRP USERNAME PRI  SIZE   RES STATE    TIME WCPU% CPU% COMMAND
   177484     177484 mirjam    20 8225M 8217M sleep    1:18  29.3 0.00 R

After upgrading from R 2.0.0 to R 2.2.1, we cannot allocate more than
about 1300 M of memory, as shown below:

 > v1 <- seq(1,2^29)

 Error: cannot allocate vector of size 2097152 Kb

 > v1 <- seq(1,2^28)
 > v2 <- seq(1,2^27)

 Error: cannot allocate vector of size 524288 Kb

 > v2 <- seq(1,2^25)
 > v3 <- seq(1,2^24)
 > v4 <- seq(1,2^23)
 > v5 <- seq(1,2^22)

 Error: cannot allocate vector of size 16384 Kb

 > v5 <- seq(1,2^21)
 > v6 <- seq(1,2^20)
 > v7 <- seq(1,2^19)
 > v8 <- seq(1,2^18)
 > q()

 Save workspace image? [y/n/c]: n
Upgrading to R 2.3.0 yields the same results.
This yields an R executable taking 1284M of RAM, refusing to allocate
more RAM, with about 30Gb free on the machine.

On Mon, 15 May 2006, m.vroonhoven at erasmusmc.nl wrote:

Dear R developers,

We have a big SGI Origin computation server with 32 cpu's and 64 Gb of
RAM. In R 2.0.0 we could run large jobs, allocating 8 Gb of RAM was not
a problem, for example by running:

v1 <- seq(1,2^29)
v2 <- seq(1,2^29)
v3 <- seq(1,2^29)
v4 <- seq(1,2^29)

This yields an R process, consuming about 8 Gb of RAM:
      PID       PGRP USERNAME PRI  SIZE   RES STATE    TIME WCPU% CPU% COMMAND
   177484     177484 mirjam    20 8225M 8217M sleep    1:18  29.3 0.00 R

After upgrading from R 2.0.0 to R 2.2.1, we cannot allocate more than
about 1300 M of memory, as shown below:

v1 <- seq(1,2^29)

 Error: cannot allocate vector of size 2097152 Kb

v1 <- seq(1,2^28)
v2 <- seq(1,2^27)

 Error: cannot allocate vector of size 524288 Kb

v2 <- seq(1,2^25)
v3 <- seq(1,2^24)
v4 <- seq(1,2^23)
v5 <- seq(1,2^22)

 Error: cannot allocate vector of size 16384 Kb

v5 <- seq(1,2^21)
v6 <- seq(1,2^20)
v7 <- seq(1,2^19)
v8 <- seq(1,2^18)
q()

 Save workspace image? [y/n/c]: n
Upgrading to R 2.3.0 yields the same results.
This yields an R executable taking 1284M of RAM, refusing to allocate
more RAM, with about 30Gb free on the machine.

You can tell if you have a 64bit build of R by looking at
.Machine$sizeof.pointer in R, which should be 8.

If not, then you need to set whatever C and Fortran compilation flags give
a 64bit system. It doesn't look to me as if R's configure script has any
special handling for C compiler flags on SGI.

If you have a 64bit build then something strange is happening. The message
you quote happens only when malloc() returns NULL, so it is hard to see
how R could be causing it, though.

You can tell if you have a 64bit build of R by looking at
.Machine$sizeof.pointer in R, which should be 8.

And if this is a 32-bit build, it is working as expected given the limited 
address space.

If not, then you need to set whatever C and Fortran compilation flags give
a 64bit system. It doesn't look to me as if R's configure script has any
special handling for C compiler flags on SGI.

Well, the R-admin manual says (under IRIX)

  @R{} 2.1.0 has been successfully built on IRIX64 6.5 using both
  @command{gcc} and the native (MipsPro 7.4) compiler. However, neither
  version has passed @command{make check} due to a problem with time
  zones (see below).  A 64-bit executable has not been successfully
  built.

so we could not use special handling for a system we have not been told 
how to build in 64-bit mode.

Here we don't know the OS, the compiler, the flags used .... and that 
definitely is a bug.

On a solaris 9 box here, gcc 3.4.2 defaults to build 32-bit binaries for some 
unknown reason,

but can be pursuaded to build 64-bit executables by passing the -m64 
switch explicitly. (this is first hand experience, discovered while 
looking into a similiar anomaly with another piece of software). I think 
gcc on early version of Mac OS X (10.2?) has

a similiar behavior also of defaulting to 32-bit unless told otherwise,
but can be pursuaded. Maybe gcc on IRIS can be similiarly pursuaded
too?

gcc -v
Reading specs from /usr/local/lib/gcc/sparc-sun-solaris2.9/3.4.2/specs
Configured with: ../configure --with-as=/usr/ccs/bin/as 
--with-ld=/usr/ccs/bin/ld --disable-nls

Mirjam van Vroonhoven wrote:

Prof Brian Ripley wrote on Mon, May 15, 2006 at 04:09:38PM +0100:

You can tell if you have a 64bit build of R by looking at
.Machine$sizeof.pointer in R, which should be 8.

And if this is a 32-bit build, it is working as expected given the limited 
address space.

It is a 32 bit build. My R 2.0.0 is a 64bit build.

If not, then you need to set whatever C and Fortran compilation flags 
give
a 64bit system. It doesn't look to me as if R's configure script has any
special handling for C compiler flags on SGI.

Well, the R-admin manual says (under IRIX)

  @R{} 2.1.0 has been successfully built on IRIX64 6.5 using both
  @command{gcc} and the native (MipsPro 7.4) compiler. However, neither
  version has passed @command{make check} due to a problem with time
  zones (see below).  A 64-bit executable has not been successfully
  built.

so we could not use special handling for a system we have not been told 
how to build in 64-bit mode.

Here we don't know the OS, the compiler, the flags used .... and that 
definitely is a bug.

The OS is irix 6.5.27, compiler = gcc, no special flags.

But that is no different from what I recall to have done when building
R 2.0.0 a long time ago.

I'll try to find out how to build a 64 bit executable, and see wether it
works. Probably that is easier using the native mipspro compiler.
Will report back to you guys when I find out a way to build a working
64bit R 2.3.0 on SGI/irix64

Thanks for the time,

                                                Mirjam

Prof Brian Ripley wrote on Mon, May 15, 2006 at 04:09:38PM +0100:

You can tell if you have a 64bit build of R by looking at
.Machine$sizeof.pointer in R, which should be 8.

And if this is a 32-bit build, it is working as expected given the limited 
address space.

It is a 32 bit build. My R 2.0.0 is a 64bit build.

If not, then you need to set whatever C and Fortran compilation flags give
a 64bit system. It doesn't look to me as if R's configure script has any
special handling for C compiler flags on SGI.

Well, the R-admin manual says (under IRIX)

  @R{} 2.1.0 has been successfully built on IRIX64 6.5 using both
  @command{gcc} and the native (MipsPro 7.4) compiler. However, neither
  version has passed @command{make check} due to a problem with time
  zones (see below).  A 64-bit executable has not been successfully
  built.

so we could not use special handling for a system we have not been told 
how to build in 64-bit mode.

Here we don't know the OS, the compiler, the flags used .... and that 
definitely is a bug.

The OS is irix 6.5.27, compiler = gcc, no special flags.

But that is no different from what I recall to have done when building
R 2.0.0 a long time ago.

I'll try to find out how to build a 64 bit executable, and see wether it
works. Probably that is easier using the native mipspro compiler.
Will report back to you guys when I find out a way to build a working
64bit R 2.3.0 on SGI/irix64

Thanks for the time,

                                                Mirjam