Skip to content

A question on R memory management in .Fortran() calls under Windows

8 messages · Simone Giannerini, Simon Urbanek, Duncan Murdoch

Original
Simone Giannerini
# 
Dear R community,

I have a question on how R manages memory allocation in .Fortran()
calls under Windows.
In brief, apparently, it is not possible to allocate large matrices
inside a Fortran subroutine
  unless you pass them as arguments. If you do not act in this way
RGUI crashes with a stack overflow error and acting  on memory through
vsize nsize ppsize and memory.limit does not help at all.

****************************************************************************************
Details of the configurations on which I performed testing follow:
R 2.1.1 on WinXP Pro SP2 ITA 
PC1: AMD 64 3700+ 1GB RAM 
PC2: AMD AthlonXP 2400+ 512Mb RAM
Compaq Visual Fortran pro 6.6C
****************************************************************************************

To give an idea I attach a brief example on how to reproduce this:
Create two simple subroutines 'foo' and 'foobis' that, say, give the
sum of the elements of a matrix:

*** file foo.f90 starts
***********************************************************************
SUBROUTINE foo(X,M,N,S)
!DEC$ ATTRIBUTES DLLEXPORT,C,REFERENCE,ALIAS:'foo_' :: FOO

    IMPLICIT NONE
    integer:: M,N
    real*8:: X(M,N),S

    S  = sum(X);

END SUBROUTINE foo

SUBROUTINE foobis(M,N,S)
!DEC$ ATTRIBUTES DLLEXPORT,C,REFERENCE,ALIAS:'foobis_' :: FOOBIS

    IMPLICIT NONE
    integer:: M,N
    real*8:: X(M,N),S
    X = 1;
    S  = sum(X);

END SUBROUTINE foobis

*** file foo.f90 ends
***********************************************************************

Notice that the matrix X is an input argument in foo and an internal
variable in foobis.
After compiling and linking turn to R:
**************************************************************************
[1] TRUE
[1] TRUE
[1] 100
[1] 100

## no problem here with small matrices, let's increase the size
[1] 3e+05  ## OK


.Fortran("foobis",as.integer(M),as.integer(N),S=as.double(0))$S;
##  *** R GUI CRASHES WITH A STACK OVERFLOW ERROR ***
****************************************************************************
Any suggestion would be greatly appreciated, I apologize if this
problem has already been addressed previously,  I did not notice it.
 Many thanks in advance,
 
 Simone
______________________________________________________ 

Simone Giannerini
Dipartimento di Scienze Statistiche "Paolo Fortunati"
Universita' di Bologna
Via delle belle arti 41 - 40126  Bologna,  ITALY
Tel: +39 051 2098248  Fax: +39 051 232153
E-mail: giannerini at stat.unibo.it
Duncan Murdoch
#
On 9/9/2005 1:04 PM, Simone Giannerini wrote:
It looks as though your Fortran compiler is allocating the new matrix on 
the stack.  R doesn't give you a huge stack, and that's causing the 
overflow.  When you get R to do the allocation, it does it on the heap, 
which has no artificial limits.  Only a pointer to the object ends up on 
the stack.

I'd say your only reasonable workarounds are to tell your compiler to 
use the heap for the local matrix allocation (if that's possible), or do 
your allocations in R.

There might be some utility somewhere which can modify Rgui.exe to tell 
it to start with a larger stack, and there is probably some linker 
option to put one in place, but it's not really reasonable to increase 
R's stack, because in almost all other situations, writing that much 
data to the stack is a sign of an infinite recursion.

Duncan Murdoch
Simon Urbanek
# 
Simone,
On Sep 9, 2005, at 1:04 PM, Simone Giannerini wrote:

            

      
      
      
    

        
I suspect that this is a problem of your compiler, not R, because it  
works without problems for me:

 > dyn.load("foo.so")
 > M=10
 > N=10
 > X=matrix(1,M,N)
 > .Fortran("foo",X,as.integer(M),as.integer(N),S=as.double(0))$S
[1] 100
 > .Fortran("foobis",as.integer(M),as.integer(N),S=as.double(0))$S
[1] 100
 > M=3000
 > N=100
 > X=matrix(1,M,N)
 > .Fortran("foo",X,as.integer(M),as.integer(N),S=as.double(0))$S
[1] 3e+05
 > .Fortran("foobis",as.integer(M),as.integer(N),S=as.double(0))$S
[1] 3e+05
 > M=10000
 > N=10000
 > X=matrix(1,M,N)
 > .Fortran("foo",X,as.integer(M),as.integer(N),S=as.double(0))$S
[1] 1e+08
 > .Fortran("foobis",as.integer(M),as.integer(N),S=as.double(0))$S
[1] 1e+08

Tested on
PC1: Win XP SP2, AMD64 3000+, 1GB RAM, gfortran 4.1.0 (20050902)
PC2: OS X, G5 1.8, 1GB RAM, gfortran 4.0.1

Cheers,
Simon

  
  


  


      

    

    
2 days later

    

      
      

        


  

        

      Simone Giannerini
    

    

      
      #
    

  


  

      
Dear Duncan and Simon,

many thanks for your helpful reply.

            

      
      
      
    

        
yes, CVF allocates automatic objects on the stack and apparently there
is no way of changing it. By the way, increasing the stack of the
fortran process when linking does not solve the problem

            

      
      
      
    

        
I might follow the second way, in any case, I am considering switching
to Linux, I have also  considered changing compiler under Win,  any
suggestions on the choice would be welcomed.
 
Many thanks again,
kind regards,

Simone Giannerini

  
  


  


      

    

    

      
      

        


  

        

      Duncan Murdoch
    

    

      
      #
    

  


  

      
Simone Giannerini wrote:

            

      
      
      
    

        
I think it's far from the best optimizing compiler, but the Fortran that 
comes with MinGW (g77 currently in Windows) is the one used to build R, 
so it's the one that will is most likely to work with it without 
fiddling.  But I don't use Fortran, so I don't know what else is available.

Duncan Murdoch

  
  


  


      

    

    

      
      

        


  

        

      Simon Urbanek
    

    

      
      #
    

  


  

      
Simone,

        
On Sep 12, 2005, at 4:30 AM, Simone Giannerini wrote:

        

            

      
      
      
    

        
Yes, that's bad news.

            

      
      
      
    

        
In general the stack size is also governed by the system limits so  
you may need to increase those as well, but still, that won't really  
solve your problem.

            

      
      
      
    

        
As Duncan was mentioning g77 is your friend if you can convert your  
code to f77. If you don't have that option, you're partially on your  
own. GNU Fortran 95 (gfortran) may be an option as it exists both for  
unix and Windows (although not as a part of MinGW), but R currently  
doesn't provide .f90 target so you'll need to add your own small  
Makevars.

Cheers,
Simon

  
  


  


      

    

    

      
      

        


  

        

      Simone Giannerini
    

    

      
      #
    

  


  

          

      
      
      
    

        
Thanks a lot anyway,
 in a comparative review I've come through recently 
(http://www.polyhedron.co.uk/compare.html)
 the Salford compiler seems to be one of the most protected against
errors but also one of the slowest. Intel compiler seems to be one of
the fastest but not the best on protection.

Simone Giannerini

  
  


  


      

    

    

      
      

        


  

        

      Simone Giannerini
    

    

      
      #
    

  


  

      
On 9/12/05, Simon Urbanek <simon.urbanek at r-project.org> wrote:

            

      
      
      
    

        
Many thanks, I did not know about gfortran for Windows, I will have a
look at it.

Kind regards,
Simone