Skip to content

memory allocation problems

8 messages · Antonio P. Ramos, Sean Davis, Kasper Daniel Hansen +2 more

Original
Antonio P. Ramos
# 
Hi everybody,

I have a memory allocation problem while using  R in my macbook pro,
which runs the latest leopard. I'm trying to run a monte carlo
simulation with 500,000 interactions, but the machine failed:


Starting MCMC Iterations...
Error: cannot allocate vector of size 1.2 Gb
R(176,0xa0640fa0) malloc: *** mmap(size=1239990272) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
R(176,0xa0640fa0) malloc: *** mmap(size=1239990272) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug


Since my machine has 4 Gb of memory, and since I'm not running nothing
in addition to the simulation, I found it strange. This is my machine:

 Model Identifier:     MacBookPro3,1
 Processor Name:       Intel Core 2 Duo
 Processor Speed:      2.4 GHz
 Memory:       4 GB


Unfortunately, I could figure it out how to solve it. Any help?


Many thanks in advance,

Antonio Ramos.
Sean Davis
# 
On Sun, Jun 29, 2008 at 6:35 AM, Antonio P. Ramos
<ramos.grad.student at gmail.com> wrote:
The error message above means that R failed to allocate a vector of
size 1.2Gb.  That doesn't mean that R was using only 1.2 Gb, but that
it was trying to allocate a new block of memory of that size in
addition to the memory that was already in use.  The system on the Mac
uses a fair amount of memory; R was probably using memory as well.  In
short, you probably need more memory or be more clever about how you
are using the memory you have.  Without more details about what you
are doing, it is difficult to know how to change the latter.

Sean
Kasper Daniel Hansen
# 
Like Sean is aying, you most likely are using _way_ more memory than  
1.2 GB.

However, if you a re running 32bit R (which is the case if you use the  
CRAN binary) R can only access 2GB, so you can squeeze a little more  
out of your machine by switching to a 64bit version of R. You can  
check what version you have by typing
R> .Machine
and look for sizeof.pointer - if it is 4 you are using 32bit, if it is  
8 you are using 64 bit.

If you want the 64 bit version of R you can download a binary from  
Simon's page: r.research.att.com , but you need to also get the  
preview build of GCC 4.2 which is available from Apple's developer  
site (although hard to find these days).

Kasper
On Jun 30, 2008, at 3:23 AM, Sean Davis wrote:

            

      
      
      
    

        

      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Simon Urbanek
    

    

      
      #
    

  


  

      
On Jun 30, 2008, at 1:04 PM, Kasper Daniel Hansen wrote:

        

            

      
      
      
    

        
That's not true, 32-bit process can use up to about 3.5GB of RAM:

Virtual Memory Map of process 2849 (R)
Output report format:  2.2  -- 32-bit process
[...]
ReadOnly portion of Libraries: Total=72.9M resident=36.6M(50%)  
swapped_out_or_unallocated=36.3M(50%)
Writable regions: Total=3.4G written=3.4G(100%) resident=3.4G(99%)  
swapped_out=3352K(0%) unallocated=19.3M(1%)

so it should make no real difference for Antonio (unless he doesn't  
mind waiting while the machine swaps). Nonetheless using 64-bit R is  
fine as well, especially on Leopard - albeit that doesn't fix  
incorrect use of memory by users :).

Cheers,
S

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Kasper Daniel Hansen
    

    

      
      #
    

  


  

      
Thanks for the clarification. How did you get that output?

Kasper

        
On Jun 30, 2008, at 10:23 AM, Simon Urbanek wrote:

        

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Antonio P. Ramos
    

    

      
      #
    

  


  

      
Thanks for the comments.

What I'm doing is very simple: I'm running an one dimensional item
response model, similar to the ones use in psychology and educational
testing data via Markov Chain Monte Carlo Methods.


model_m12<- ideal(rollcall_m2,  maxiter = 500 000 000, thin = 1000,
burnin = 5000,
		   store.item = TRUE,  normalize=T,
priors=list(xp=1e-12,xpv=1e-12,bp=1e-12,bpv=1e-12), verbose=T)

# my data matrix is provided by the rollcall object, but it has just
155*17 dimensions: rollcall_m2

# the number of interactions is maxiter/thin= 500,000

# store.item=true is the main source of the problem: it's store the
discrimination
# parameter, which consumer a large amount of memory. Unfortunately, I
need this information.


So, if R can access up to 3.5 Gb how can I fix the problem. I'm sure
lot's of mac users will be also interested in increasing its memory
allocation capabilities in R.

Cheers,

Antonio.





On Mon, Jun 30, 2008 at 11:38 AM, Kasper Daniel Hansen

        
<khansen at stat.berkeley.edu> wrote:

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

            

      
      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Kasper Daniel Hansen
    

    

      
      #
    

  


  

      
On Jun 30, 2008, at 12:07 PM, Antonio P. Ramos wrote:

        

            

      
      
      
    

        
It works out of the box (that means you do not have to do anything).  
The fact that you get an out of memory error just means that your R  
session is using more than 3.4 GB of RAM. Think of it in the following  
way: say you are using 3.3 GB of RAM and then you create a new object  
with a size of 0.2GB which takes you over the limit. Than you would  
get an error of "cannot allocate 0.2 GB of RAM". So the error you get  
does not mean that you cannot use more than 1.1 GB. It just means you  
went over the 3.4 GB limit.

Now, from your description it sounds like you should be able to do  
your simulation within a memory usage of 3.4 GB. But whether or not  
that is possible depends crucially on the implementation which we have  
no idea about. There are many naive ways to implement this in R that  
will take you far beyond the memory limit. For example you are doing  
thinning. But are you for example first computing the MCM chain  
without thinning and then thinning afterwards - then thinning will not  
help you in your memory consumption.

I would spend some time checking up on the implementation of the  
sampler you are using.

Kasper

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

            

      
      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Thomas Lumley
    

    

      
      #
    

  


  

      
It might be worth reminding people that gc() reports the maximum (R heap) 
memory use as well as current memory use.

The memory profiler might also be useful for tracking down what is 
happening, but I don't think it's compiled into the CRAN binary of R.

 	-thomas

        
On Mon, 30 Jun 2008, Kasper Daniel Hansen wrote:

        

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

        
Thomas Lumley			Assoc. Professor, Biostatistics
tlumley at u.washington.edu	University of Washington, Seattle