Skip to content

Memory leak in R v1.5.1?

4 messages · Jan Meyer, Martin Maechler, Adaikalavan Ramasamy

#
Hi,

I am trying to minimize a rather complex function of 5 parameters with 
gafit and nlm. Besides some problems with both optimization algorithms 
(with respect to consistantly generating similar results), I tried to 
run this optimization about a hundred times for yet two other parameters.

Unfortunately, as the log below shows, during that batch process R 
starts to eat up all my RAM, preventing me to simply run this job over 
night.

The problem seems to be related to memory used by cons cells, which are 
  internal data structures like parse trees etc. as I understand. (For 
more see memory.profile() call all the way below) Despite me forcing 
garbage collection between each optimization run, these cells accumulate 
quite rapidly.

All I am doing is calling my function (implemented as an R function 
calling another R function) millions of times through gafit() and nlm(). 
The little data that my function needs is loaded into the environment 
prior to optimization. Either algorithm seems to accumulate cons cells.

Any idea what could be the cause? My code maybe? Is this a memory leak 
within R?

Is there any other way to force garbage collection (or something 
equivalent) on the ncells?

Thanks.

- Jan

P.S. I run R v1.5.1 on i386 SuSE Linux v7.3 using the precompiled 
binaries from CTAN for SuSE v7.3.

-----------------
   > fName ="designOpt"; Sweave(paste(fName, "rnw", sep="."),
driver=Rtangle()); sourc
e(paste(fName, "R", sep="."), local = TRUE)
Writing to file designOpt.R
4 : designOpt-designCollectionFileFuncs.R
hybridDesignOpt() called with: weight.margin= 0 , KG.margin= 0
GA Design: ShipMathModel( LWL=551.205711406606, B=45.4637940845938,
Cp=0.7198803735
31636, Cx=0.837682085129617, W.BP=872.396799843704,
H.DKh=12.0780590098038, H.DKd=8
.25941049673737 ) = (Cost=3.16 , Err.Pwr=15.0, Err.Range=18.5, B/H=3.55)
Garbage collection 170 = 160+7+3 (level 2) ...
147999 cons cells free (26%)
7.8 Mbytes of heap free (77%)
            used (Mb) gc trigger (Mb) limit (Mb)
Ncells 417001 11.2     565000 15.1       16.1
Vcells 291029  2.3    1310720 10.0       20.0
                total       used       free     shared    buffers     cached
Mem:        124948      93736      31212          0      17308      25824
-/+ buffers/cache:      50604      74344
Swap:       265032          8     265024
[1] 0
Gradient Design: ShipMathModel( LWL=537.059747125118,
B=44.8171342153699, Cp=0.7222
67017738667, Cx=0.836634804176116, W.BP=872.197302196268,
H.DKh=12.0780590098038, H
.DKd=8.14724020090996 ) = (Cost=3.15  , Err.Pwr=11.5, Err.Range=25.1,
B/H=3.45)
Garbage collection 184 = 172+8+4 (level 2) ...
141287 cons cells free (25%)
7.7 Mbytes of heap free (77%)
            used (Mb) gc trigger (Mb) limit (Mb)
Ncells 423713 11.4     565000 15.1       16.1
Vcells 299539  2.3    1310720 10.0       20.0
                total       used       free     shared    buffers     cached
Mem:        124948      93736      31212          0      17308      25824
-/+ buffers/cache:      50604      74344
Swap:       265032          8     265024
[1] 0
   >>>>>>    im =  2 ; time =  Sat Aug  3 15:37:34 2002
hybridDesignOpt() called with: weight.margin= 0 , KG.margin= 1.274
GA Design: ShipMathModel( LWL=531.251923358073, B=44.9592165303293,
Cp=0.726323984032877, Cx=0.823656217446154, W.BP=779.115652389105,
H.DKh=11.7190083082315, H.DKd=8.55853079052759 ) = (Cost=3.16  ,
Err.Pwr=13.7, Err.Range=14.3, B/H=3.47)
Garbage collection 355 = 262+46+47 (level 2) ...
75179 cons cells free (13%)
7.1 Mbytes of heap free (70%)
            used (Mb) gc trigger (Mb) limit (Mb)
Ncells 489821 13.1     565000 15.1       16.1
Vcells 383011  3.0    1310720 10.0       20.0
                total       used       free     shared    buffers     cached
Mem:        124948      93736      31212          0      17308      25824
-/+ buffers/cache:      50604      74344
Swap:       265032          8     265024
[1] 0
Gradient Design: ShipMathModel( LWL=531.076590422056,
B=44.9466118426414, Cp=0.726049001140884, Cx=0.823594459634168,
W.BP=779.10512867677, H.DKh=11.718702931698, H.DKd=8.55702887701944 ) =
(Cost=3.16  , Err.Pwr=13.4, Err.Range=14.6, B/H=3.46)
Garbage collection 373 = 268+51+54 (level 2) ...
70538 cons cells free (12%)
7.0 Mbytes of heap free (70%)
            used (Mb) gc trigger (Mb) limit (Mb)
Ncells 494462 13.3     565000 15.1       16.1
Vcells 388922  3.0    1310720 10.0       20.0
                total       used       free     shared    buffers     cached
Mem:        124948      93736      31212          0      17308      25824
-/+ buffers/cache:      50604      74344
Swap:       265032          8     265024
[1] 0
   >>>>>>    im =  3 ; time =  Sat Aug  3 15:40:14 2002
hybridDesignOpt() called with: weight.margin= 0 , KG.margin= 2.548
GA Design: ShipMathModel( LWL=578.21472911227, B=49.466359186738,
Cp=0.702282574662
H.DKd=11.5397975180451 ) = (Cost=3.25  , Err.Pwr=0.8, Err.Range=-1.5,
B/H=3.28)
Garbage collection 636 = 358+130+148 (level 2) ...
4432 cons cells free (0%)
6.4 Mbytes of heap free (63%)
            used (Mb) gc trigger (Mb) limit (Mb)
Ncells 560568 15.0     565000 15.1       16.1
Vcells 472355  3.7    1310720 10.0       20.0
                total       used       free     shared    buffers     cached
Mem:        124948      94624      30324          0      17308      25824
-/+ buffers/cache:      51492      73456
Swap:       265032          8     265024
[1] 0
Error: cons memory exhausted (limit reached?)
In addition: There were 24 warnings (use warnings() to see them)
Lost warning messages
   > gc()
            used (Mb) gc trigger (Mb) limit (Mb)
Ncells 563691 15.1     565000 15.1       16.1
Vcells 476605  3.7    1310720 10.0       20.0
   > rm(list = ls(all = TRUE))
   > gc()
            used (Mb) gc trigger (Mb) limit (Mb)
Ncells 554338 14.9     565000 15.1       16.1
Vcells 467858  3.6    1310720 10.0       20.0
  > rm(list = ls(all = TRUE))
  > gc()
           used (Mb) gc trigger (Mb) limit (Mb)
Ncells 554338 14.9     565000 15.1       16.1
Vcells 467858  3.6    1310720 10.0       20.0
  > memory.profile()
      NILSXP     SYMSXP    LISTSXP     CLOSXP     ENVSXP    PROMSXP
LANGSXP
           1       5031     264110       1401      16047          0
60670
SPECIALSXP BUILTINSXP    CHARSXP     LGLSXP                           INTSXP
          59        503      29179       1682          0          0
     24
     REALSXP    CPLXSXP     STRSXP     DOTSXP     ANYSXP     VECSXP
EXPRSXP
      135616          8      23982          0          0      16053
      0
              EXTPTRSXP WEAKREFSXP
           0          0          0


-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To: r-help-request at stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._
#
Jan> Hi,

    Jan> I am trying to minimize a rather complex function of 5
    Jan> parameters with gafit and nlm. Besides some problems
    Jan> with both optimization algorithms (with respect to
    Jan> consistantly generating similar results), I tried to
    Jan> run this optimization about a hundred times for yet two
    Jan> other parameters.

    Jan> Unfortunately, as the log below shows, during that
    Jan> batch process R starts to eat up all my RAM, preventing
    Jan> me to simply run this job over night.

    Jan> The problem seems to be related to memory used by cons
    Jan> cells, which are internal data structures like parse
    Jan> trees etc. as I understand. (For more see
    Jan> memory.profile() call all the way below) Despite me
    Jan> forcing garbage collection between each optimization
    Jan> run, these cells accumulate quite rapidly.

which must be a leak bug somewhere, yes, 
not necessarily in "core R" though.  It could be in your code or
in gafit() {which is from a contributed package and hence not in
"core R"}.

    Jan> All I am doing is calling my function (implemented as an R function 
    Jan> calling another R function) millions of times through gafit() and nlm(). 
    Jan> The little data that my function needs is loaded into the environment 
    Jan> prior to optimization. Either algorithm seems to accumulate cons cells.

    Jan> Any idea what could be the cause? My code maybe? Is this a memory leak 
    Jan> within R?

both are possible.  We have had rare cases of memory leaks
{found and fixed} in the past.

But since you mention "1.5.1" in the subject, have you used your
code in earlier versions of R and not seen the memory growth?

    Jan> Is there any other way to force garbage collection (or something 
    Jan> equivalent) on the ncells?

    Jan> Thanks.

Rather we need vastly better readable example code in order to
tell. The stuff you append looks terribly garbled (by your
e-mail software probably?), and what we really need is
*reproducible* source code (i.e. data, function, all available
in that R script) rather than a session log.
If it is true that you only have a basic function and everything
else is nlm() and gafit() calls, such a script shouldn't be too
long.

Thank you for the report and (in advance) for such a more useful
script.

Martin Maechler <maechler at stat.math.ethz.ch>	http://stat.ethz.ch/~maechler/
Seminar fuer Statistik, ETH-Zentrum  LEO C16	Leonhardstr. 27
ETH (Federal Inst. Technology)	8092 Zurich	SWITZERLAND
phone: x-41-1-632-3408		fax: ...-1228			<><



    Jan> - Jan

    Jan> P.S. I run R v1.5.1 on i386 SuSE Linux v7.3 using the precompiled 
    Jan> binaries from CTAN for SuSE v7.3.

    Jan> -----------------
    >> fName ="designOpt"; Sweave(paste(fName, "rnw", sep="."),
    Jan> driver=Rtangle()); sourc
    Jan> e(paste(fName, "R", sep="."), local = TRUE)
    Jan> Writing to file designOpt.R
    Jan> 4 : designOpt-designCollectionFileFuncs.R
    Jan> hybridDesignOpt() called with: weight.margin= 0 , KG.margin= 0
    Jan> GA Design: ShipMathModel( LWL=551.205711406606, B=45.4637940845938,
    Jan> Cp=0.7198803735
    Jan> 31636, Cx=0.837682085129617, W.BP=872.396799843704,
    Jan> H.DKh=12.0780590098038, H.DKd=8
    Jan> .25941049673737 ) = (Cost=3.16 , Err.Pwr=15.0, Err.Range=18.5, B/H=3.55)
    Jan> Garbage collection 170 = 160+7+3 (level 2) ...
    Jan> 147999 cons cells free (26%)
    Jan> 7.8 Mbytes of heap free (77%)
    Jan> used (Mb) gc trigger (Mb) limit (Mb)
    Jan> Ncells 417001 11.2     565000 15.1       16.1
    Jan> Vcells 291029  2.3    1310720 10.0       20.0
    Jan> total       used       free     shared    buffers     cached
    Jan> Mem:        124948      93736      31212          0      17308      25824
    Jan> -/+ buffers/cache:      50604      74344
    Jan> Swap:       265032          8     265024
    Jan> [1] 0
    Jan> Gradient Design: ShipMathModel( LWL=537.059747125118,
    Jan> B=44.8171342153699, Cp=0.7222
    Jan> 67017738667, Cx=0.836634804176116, W.BP=872.197302196268,
    Jan> H.DKh=12.0780590098038, H
    Jan> .DKd=8.14724020090996 ) = (Cost=3.15  , Err.Pwr=11.5, Err.Range=25.1,
    Jan> B/H=3.45)
    Jan> Garbage collection 184 = 172+8+4 (level 2) ...
    Jan> 141287 cons cells free (25%)
    Jan> 7.7 Mbytes of heap free (77%)
    Jan> used (Mb) gc trigger (Mb) limit (Mb)
    Jan> Ncells 423713 11.4     565000 15.1       16.1
    Jan> Vcells 299539  2.3    1310720 10.0       20.0
    Jan> total       used       free     shared    buffers     cached
    Jan> Mem:        124948      93736      31212          0      17308      25824
    Jan> -/+ buffers/cache:      50604      74344
    Jan> Swap:       265032          8     265024
    Jan> [1] 0
    >>>>>>> im =  2 ; time =  Sat Aug  3 15:37:34 2002
    Jan> hybridDesignOpt() called with: weight.margin= 0 , KG.margin= 1.274
    Jan> GA Design: ShipMathModel( LWL=531.251923358073, B=44.9592165303293,
    Jan> Cp=0.726323984032877, Cx=0.823656217446154, W.BP=779.115652389105,
    Jan> H.DKh=11.7190083082315, H.DKd=8.55853079052759 ) = (Cost=3.16  ,
    Jan> Err.Pwr=13.7, Err.Range=14.3, B/H=3.47)
    Jan> Garbage collection 355 = 262+46+47 (level 2) ...
    Jan> 75179 cons cells free (13%)
    Jan> 7.1 Mbytes of heap free (70%)
    Jan> used (Mb) gc trigger (Mb) limit (Mb)
    Jan> Ncells 489821 13.1     565000 15.1       16.1
    Jan> Vcells 383011  3.0    1310720 10.0       20.0
    Jan> total       used       free     shared    buffers     cached
    Jan> Mem:        124948      93736      31212          0      17308      25824
    Jan> -/+ buffers/cache:      50604      74344
    Jan> Swap:       265032          8     265024
    Jan> [1] 0
    Jan> Gradient Design: ShipMathModel( LWL=531.076590422056,
    Jan> B=44.9466118426414, Cp=0.726049001140884, Cx=0.823594459634168,
    Jan> W.BP=779.10512867677, H.DKh=11.718702931698, H.DKd=8.55702887701944 ) =
    Jan> (Cost=3.16  , Err.Pwr=13.4, Err.Range=14.6, B/H=3.46)
    Jan> Garbage collection 373 = 268+51+54 (level 2) ...
    Jan> 70538 cons cells free (12%)
    Jan> 7.0 Mbytes of heap free (70%)
    Jan> used (Mb) gc trigger (Mb) limit (Mb)
    Jan> Ncells 494462 13.3     565000 15.1       16.1
    Jan> Vcells 388922  3.0    1310720 10.0       20.0
    Jan> total       used       free     shared    buffers     cached
    Jan> Mem:        124948      93736      31212          0      17308      25824
    Jan> -/+ buffers/cache:      50604      74344
    Jan> Swap:       265032          8     265024
    Jan> [1] 0
    >>>>>>> im =  3 ; time =  Sat Aug  3 15:40:14 2002
    Jan> hybridDesignOpt() called with: weight.margin= 0 , KG.margin= 2.548
    Jan> GA Design: ShipMathModel( LWL=578.21472911227, B=49.466359186738,
    Jan> Cp=0.702282574662
    Jan> H.DKd=11.5397975180451 ) = (Cost=3.25  , Err.Pwr=0.8, Err.Range=-1.5,
    Jan> B/H=3.28)
    Jan> Garbage collection 636 = 358+130+148 (level 2) ...
    Jan> 4432 cons cells free (0%)
    Jan> 6.4 Mbytes of heap free (63%)
    Jan> used (Mb) gc trigger (Mb) limit (Mb)
    Jan> Ncells 560568 15.0     565000 15.1       16.1
    Jan> Vcells 472355  3.7    1310720 10.0       20.0
    Jan> total       used       free     shared    buffers     cached
    Jan> Mem:        124948      94624      30324          0      17308      25824
    Jan> -/+ buffers/cache:      51492      73456
    Jan> Swap:       265032          8     265024
    Jan> [1] 0
    Jan> Error: cons memory exhausted (limit reached?)
    Jan> In addition: There were 24 warnings (use warnings() to see them)
    Jan> Lost warning messages
    >> gc()
    Jan> used (Mb) gc trigger (Mb) limit (Mb)
    Jan> Ncells 563691 15.1     565000 15.1       16.1
    Jan> Vcells 476605  3.7    1310720 10.0       20.0
    >> rm(list = ls(all = TRUE))
    >> gc()
    Jan> used (Mb) gc trigger (Mb) limit (Mb)
    Jan> Ncells 554338 14.9     565000 15.1       16.1
    Jan> Vcells 467858  3.6    1310720 10.0       20.0
    >> rm(list = ls(all = TRUE))
    >> gc()
    Jan> used (Mb) gc trigger (Mb) limit (Mb)
    Jan> Ncells 554338 14.9     565000 15.1       16.1
    Jan> Vcells 467858  3.6    1310720 10.0       20.0
    >> memory.profile()
    Jan> NILSXP     SYMSXP    LISTSXP     CLOSXP     ENVSXP    PROMSXP
    Jan> LANGSXP
    Jan> 1       5031     264110       1401      16047          0
    Jan> 60670
    Jan> SPECIALSXP BUILTINSXP    CHARSXP     LGLSXP                           INTSXP
    Jan> 59        503      29179       1682          0          0
    Jan> 24
    Jan> REALSXP    CPLXSXP     STRSXP     DOTSXP     ANYSXP     VECSXP
    Jan> EXPRSXP
    Jan> 135616          8      23982          0          0      16053
    Jan> 0
    Jan> EXTPTRSXP WEAKREFSXP
    Jan> 0          0          0
-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To: r-help-request at stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._
#
Moin Martin,

danke fuer Deine schnelle Antwort.

[I will continue in English for the sake of forwardability]

It's very understandable to me that you're asking for readable example 
code. But then I was wondering, do you really want to look through ~1300 
lines of code (w/o comments) generated from a collection of Sweave 
files? I guess in that case I'd have to reduce my problem down to a 
level that can be easily overlooked but still causes the problem. 
Unfortunately I don't have the time for that during the next two weeks 
(really - please don't ask).

My idea was to ask for some hints from you first, then try to fix it my 
own, if that doesn't work quickly, find some ugly work around and pick 
up the problem in two weeks for a real solution.

For example I don't understand how it is possible for my code to 
generate this memory leak in particular, since this is not me playing 
the pointer wizard in C and since it's a problem with cons cells and not 
the heap. If I know what I could be doing wrong, I might be able to 
solve the problem myself.

Should the consumption of cons cell ever grow during the course of a 
session? [The consumption of these cells does not even go down if I 
terminate the script.] I force garbage collection with gc() does that 
also take care of the cons cell or heap only?

Thanks.

- Jan
Martin Maechler wrote:
-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To: r-help-request at stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._
#
Hi ! I am trying to minimise a function over 2 variables simultaneously. Is
this possible ? I have only recently been introduced to optim() and
optimise() function. Here is simple example of what I tried to do :

silly.fn <- function(x,y) { (x-5)^2 + (y-2)^2 }
optim( par=c(3,1) , silly.fn )

Analytically, the answer should be (5,2) but the computer is asking for the
y value. Is there a function to do the minimisation simultaneously in S-PLUS
or R ?

Many thanks, Adai.

-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To: r-help-request at stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._