Skip to content

R process killed when allocating too large matrix (Mac OS X)

9 messages · Marius Hofert, Uwe Ligges, Brian Ripley +4 more

Original
Marius Hofert
# 
Hi Simon,

... all interesting (but quite a bit above my head). I only read
'Linux' and want to throw in that this problem does not appear on
Linux (it seems). I talked about this with Martin Maechler and he
reported that the same example (on one of his machines; with NA_real_
instead of '0's in the matrix) gave:

  Error: cannot allocate vector of size 70.8 Gb
    Timing stopped at: 144.79 41.619 202.019

... but no killer around...

Cheers,
Marius
Uwe Ligges
#
On 05.05.2016 04:25, Marius Hofert wrote:
Well, with n=1. ;-)

Actually this also happens under Linux and I had my R processes killed 
more than once (and much worse also other processes so that we had to 
reboot a server, essentially). That's why we use job scheduling on 
servers for R nowadays ...

Best,
Uwe
Brian Ripley
#
On 05/05/2016 10:11, Uwe Ligges wrote:
Yes, Linux does not deal safely with running out of memory, although it 
is better than it was.  In my experience, only commercial Unices do that 
gracefully.

Have you tried setting a (virtual) memory limit on the process using the 
shell it is launched from?  I have found that to be effective on most 
OSes, at least in protecting other processes from being killed. 
However, some things do reserve excessive amounts of VM that they do not 
use and so cannot be run under a sensible limit.
4 days later
Jeroen Ooms
#
On 05/05/2016 10:11, Uwe Ligges wrote:
I found that setting RLIMIT_AS [1] works very well on Linux. But this
requires that you cap memory to some fixed value.
Error: cannot allocate vector of size 7.5 Gb

The RAppArmor package has many other utilities to protect your server
such from a mis-behaving process such as limiting cpu time
(RLIMIT_CPU), fork bombs (RLIMIT_NPROC) and file sizes (RLIMIT_FSIZE).

[1] http://linux.die.net/man/2/getrlimit
1 day later
Kirill Müller
# 
My ulimit package exposes this API ([1], should finally submit it to 
CRAN); unfortunately this very API seems to be unsupported on OS X 
[2,3]. Last time I looked into it, neither of the documented settings 
achieved the desired effect.


-Kirill


[1] http://krlmlr.github.io/ulimit
[2] 
http://stackoverflow.com/questions/3274385/how-to-limit-memory-of-a-os-x-program-ulimit-v-neither-m-are-working
[3] 
https://developer.apple.com/library/ios/documentation/System/Conceptual/ManPages_iPhoneOS/man2/getrlimit.2.html
On 10.05.2016 01:08, Jeroen Ooms wrote:
Martin Maechler
# 
> My ulimit package exposes this API ([1], should finally submit it to 
    > CRAN); unfortunately this very API seems to be unsupported on OS X 
    > [2,3]. Last time I looked into it, neither of the documented settings 
    > achieved the desired effect.

    > -Kirill

    > [1] http://krlmlr.github.io/ulimit
    > [2] 
    > http://stackoverflow.com/questions/3274385/how-to-limit-memory-of-a-os-x-program-ulimit-v-neither-m-are-working
    > [3] 
    > https://developer.apple.com/library/ios/documentation/System/Conceptual/ManPages_iPhoneOS/man2/getrlimit.2.html
> On 10.05.2016 01:08, Jeroen Ooms wrote:
>> On 05/05/2016 10:11, Uwe Ligges wrote:
>>> Actually this also happens under Linux and I had my R processes killed
    >>> more than once (and much worse also other processes so that we had to
    >>> reboot a server, essentially).

I agree that Linux is not consistently fine here either.

    >> I found that setting RLIMIT_AS [1] works very well on Linux. But this
    >> requires that you cap memory to some fixed value.

conceivably one could set a default cap, using something
equivalent to the data in

		sfsmisc::Sys.meminfo()
or		sfsmisc::Sys.memGB()		

(very simple, > 10 year old interfaces, based on the  Linux-only (?)
 '/proc/*' filesystem).


In an ideal word, some of us,
    from R core, Jeroen, Kyrill, <one person knowing much about Windows>,
    maintainer("microbenchmark>, ...
would sit together and devise an R function interface (based on
low level platform specific interfaces, specifically for at least
Linux/POSIX-compliant, Mac, and Windows) which would allow
something  like your rlimit(..) calls below.

We'd really need something to work on all platforms ideally,
to be used by R package maintainers
and possibly even better by R itself at startup, setting a
reasonable memory cap - which the user could raise even to +Inf (or lower
even more).

Martin

---------
Final notes about RAppArmor, not relevant to the main thread topic :

Note: I'm working in pretty well maintained Fedora Linux env,
but Apparmor is not only not activated, but even not available.
OTOH, usting  RLIMIT / gerlimit on Linux is very generally
available.

As a consequence, the three last lines of

   > require(RAppArmor)
   Loading required package: RAppArmor
   Loading required package: tools
   Failed to lookup process confinement:
   AppArmor not supported on this system
   Have a look at: sudo aa-status

were very confusing to me:  My conclusion was I could not use
the RAppArmor package.

(But that's wrong: For the  rlimit*()  functions below, one do
 *NOT* need an AppArmor-enabled version of Linux !)
		  
    >>> library(RAppArmor)
    >>> rlimit_as(1e9)
    >>> rnorm(1e9)
    >> Error: cannot allocate vector of size 7.5 Gb
    >> 
    >> The RAppArmor package has many other utilities to protect your server
    >> such from a mis-behaving process such as limiting cpu time
    >> (RLIMIT_CPU), fork bombs (RLIMIT_NPROC) and file sizes (RLIMIT_FSIZE).
    >> 
    >> [1] http://linux.die.net/man/2/getrlimit

and from my current explorations I gather that all of these are
*not* Apparmor related... so could/should maybe rather migrate
into a lightweight package not mentioning AppArmor ?
Jeroen Ooms
# 
On Thu, May 12, 2016 at 9:51 AM, Martin Maechler
<maechler at stat.math.ethz.ch> wrote:
Yes, it is a relatively recent (unadvertised) feature that the package
now builds on linux systems without libapparmor. I agree this names
the package name confusing. I'll make at least that warning more
informative.

Some background: When I started the package (5 years ago) I expected
that soon all linux distributions would have the apparmor module which
has been in the kernel since 2.6.36. However Redhat is explicitly
disabling it because they are pushing a competing MAC system (SELinux)
which they develop together with the NSA, and they really want you to
use this instead (and only this!).
I agree, it has been on the to do list for a while; Kirill and me were
talking yesterday about what would be the best route to take:

 - A small package with only the rlimit bindings
 - or: A 'linux' package with bindings to anything in the kernel,
including rlimit, but possibly other system tools.
 - or: A package targeting POSIX/unix with standard functionality that
is also available on OSX/BSD.
virgo
# 
On Thu, 12 May 2016 12:07:29 +0200
Jeroen Ooms <jeroen.ooms at stat.ucla.edu> wrote:

            

      
      
      
    

        
Maybe not so useless after reading [1] about computationally
querying the OSes on the available memory and [2] about pushing the
OSes' limits. The latter page is part of a series where each topic is
valuable on its own.

[1]?https://msdn.microsoft.com/en-us/library/aa366778.aspx
[2]?https://blogs.technet.microsoft.com/markrussinovich/2008/07/21/pushing-the-limits-of-windows-physical-memory/

            

      
      
      
    

        

      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Kirill Müller
    

    

      
      #
    

  


  

      
On 12.05.2016 09:51, Martin Maechler wrote:

            

      
      
      
    

        
I haven't found a Windows API that allows limiting the address space, 
only one that limits the working set size; it seems likely that this is 
the best we can get on OS X, too, but then my experience with OS X is 
very limited.

mallinfo() is used on Windows and seems to be available on Linux, too, 
but not on OS X.


-Kirill