Memory leak in R v1.5.1?
Moin Martin, danke fuer Deine schnelle Antwort. [I will continue in English for the sake of forwardability] It's very understandable to me that you're asking for readable example code. But then I was wondering, do you really want to look through ~1300 lines of code (w/o comments) generated from a collection of Sweave files? I guess in that case I'd have to reduce my problem down to a level that can be easily overlooked but still causes the problem. Unfortunately I don't have the time for that during the next two weeks (really - please don't ask). My idea was to ask for some hints from you first, then try to fix it my own, if that doesn't work quickly, find some ugly work around and pick up the problem in two weeks for a real solution. For example I don't understand how it is possible for my code to generate this memory leak in particular, since this is not me playing the pointer wizard in C and since it's a problem with cons cells and not the heap. If I know what I could be doing wrong, I might be able to solve the problem myself. Should the consumption of cons cell ever grow during the course of a session? [The consumption of these cells does not even go down if I terminate the script.] I force garbage collection with gc() does that also take care of the cons cell or heap only? Thanks. - Jan
Martin Maechler wrote:
"Jan" == Jan Meyer <janm at mit.edu> writes:
Jan> Hi,
Jan> I am trying to minimize a rather complex function of 5
Jan> parameters with gafit and nlm. Besides some problems
Jan> with both optimization algorithms (with respect to
Jan> consistantly generating similar results), I tried to
Jan> run this optimization about a hundred times for yet two
Jan> other parameters.
Jan> Unfortunately, as the log below shows, during that
Jan> batch process R starts to eat up all my RAM, preventing
Jan> me to simply run this job over night.
Jan> The problem seems to be related to memory used by cons
Jan> cells, which are internal data structures like parse
Jan> trees etc. as I understand. (For more see
Jan> memory.profile() call all the way below) Despite me
Jan> forcing garbage collection between each optimization
Jan> run, these cells accumulate quite rapidly.
which must be a leak bug somewhere, yes,
not necessarily in "core R" though. It could be in your code or
in gafit() {which is from a contributed package and hence not in
"core R"}.
Jan> All I am doing is calling my function (implemented as an R function
Jan> calling another R function) millions of times through gafit() and nlm().
Jan> The little data that my function needs is loaded into the environment
Jan> prior to optimization. Either algorithm seems to accumulate cons cells.
Jan> Any idea what could be the cause? My code maybe? Is this a memory leak
Jan> within R?
both are possible. We have had rare cases of memory leaks
{found and fixed} in the past.
But since you mention "1.5.1" in the subject, have you used your
code in earlier versions of R and not seen the memory growth?
Jan> Is there any other way to force garbage collection (or something
Jan> equivalent) on the ncells?
Jan> Thanks.
Rather we need vastly better readable example code in order to
tell. The stuff you append looks terribly garbled (by your
e-mail software probably?), and what we really need is
*reproducible* source code (i.e. data, function, all available
in that R script) rather than a session log.
If it is true that you only have a basic function and everything
else is nlm() and gafit() calls, such a script shouldn't be too
long.
Thank you for the report and (in advance) for such a more useful
script.
Martin Maechler <maechler at stat.math.ethz.ch> http://stat.ethz.ch/~maechler/
Seminar fuer Statistik, ETH-Zentrum LEO C16 Leonhardstr. 27
ETH (Federal Inst. Technology) 8092 Zurich SWITZERLAND
phone: x-41-1-632-3408 fax: ...-1228 <><
Jan> - Jan
Jan> P.S. I run R v1.5.1 on i386 SuSE Linux v7.3 using the precompiled
Jan> binaries from CTAN for SuSE v7.3.
Jan> -----------------
>> fName ="designOpt"; Sweave(paste(fName, "rnw", sep="."),
Jan> driver=Rtangle()); sourc
Jan> e(paste(fName, "R", sep="."), local = TRUE)
Jan> Writing to file designOpt.R
Jan> 4 : designOpt-designCollectionFileFuncs.R
Jan> hybridDesignOpt() called with: weight.margin= 0 , KG.margin= 0
Jan> GA Design: ShipMathModel( LWL=551.205711406606, B=45.4637940845938,
Jan> Cp=0.7198803735
Jan> 31636, Cx=0.837682085129617, W.BP=872.396799843704,
Jan> H.DKh=12.0780590098038, H.DKd=8
Jan> .25941049673737 ) = (Cost=3.16 , Err.Pwr=15.0, Err.Range=18.5, B/H=3.55)
Jan> Garbage collection 170 = 160+7+3 (level 2) ...
Jan> 147999 cons cells free (26%)
Jan> 7.8 Mbytes of heap free (77%)
Jan> used (Mb) gc trigger (Mb) limit (Mb)
Jan> Ncells 417001 11.2 565000 15.1 16.1
Jan> Vcells 291029 2.3 1310720 10.0 20.0
Jan> total used free shared buffers cached
Jan> Mem: 124948 93736 31212 0 17308 25824
Jan> -/+ buffers/cache: 50604 74344
Jan> Swap: 265032 8 265024
Jan> [1] 0
Jan> Gradient Design: ShipMathModel( LWL=537.059747125118,
Jan> B=44.8171342153699, Cp=0.7222
Jan> 67017738667, Cx=0.836634804176116, W.BP=872.197302196268,
Jan> H.DKh=12.0780590098038, H
Jan> .DKd=8.14724020090996 ) = (Cost=3.15 , Err.Pwr=11.5, Err.Range=25.1,
Jan> B/H=3.45)
Jan> Garbage collection 184 = 172+8+4 (level 2) ...
Jan> 141287 cons cells free (25%)
Jan> 7.7 Mbytes of heap free (77%)
Jan> used (Mb) gc trigger (Mb) limit (Mb)
Jan> Ncells 423713 11.4 565000 15.1 16.1
Jan> Vcells 299539 2.3 1310720 10.0 20.0
Jan> total used free shared buffers cached
Jan> Mem: 124948 93736 31212 0 17308 25824
Jan> -/+ buffers/cache: 50604 74344
Jan> Swap: 265032 8 265024
Jan> [1] 0
>>>>>>> im = 2 ; time = Sat Aug 3 15:37:34 2002
Jan> hybridDesignOpt() called with: weight.margin= 0 , KG.margin= 1.274
Jan> GA Design: ShipMathModel( LWL=531.251923358073, B=44.9592165303293,
Jan> Cp=0.726323984032877, Cx=0.823656217446154, W.BP=779.115652389105,
Jan> H.DKh=11.7190083082315, H.DKd=8.55853079052759 ) = (Cost=3.16 ,
Jan> Err.Pwr=13.7, Err.Range=14.3, B/H=3.47)
Jan> Garbage collection 355 = 262+46+47 (level 2) ...
Jan> 75179 cons cells free (13%)
Jan> 7.1 Mbytes of heap free (70%)
Jan> used (Mb) gc trigger (Mb) limit (Mb)
Jan> Ncells 489821 13.1 565000 15.1 16.1
Jan> Vcells 383011 3.0 1310720 10.0 20.0
Jan> total used free shared buffers cached
Jan> Mem: 124948 93736 31212 0 17308 25824
Jan> -/+ buffers/cache: 50604 74344
Jan> Swap: 265032 8 265024
Jan> [1] 0
Jan> Gradient Design: ShipMathModel( LWL=531.076590422056,
Jan> B=44.9466118426414, Cp=0.726049001140884, Cx=0.823594459634168,
Jan> W.BP=779.10512867677, H.DKh=11.718702931698, H.DKd=8.55702887701944 ) =
Jan> (Cost=3.16 , Err.Pwr=13.4, Err.Range=14.6, B/H=3.46)
Jan> Garbage collection 373 = 268+51+54 (level 2) ...
Jan> 70538 cons cells free (12%)
Jan> 7.0 Mbytes of heap free (70%)
Jan> used (Mb) gc trigger (Mb) limit (Mb)
Jan> Ncells 494462 13.3 565000 15.1 16.1
Jan> Vcells 388922 3.0 1310720 10.0 20.0
Jan> total used free shared buffers cached
Jan> Mem: 124948 93736 31212 0 17308 25824
Jan> -/+ buffers/cache: 50604 74344
Jan> Swap: 265032 8 265024
Jan> [1] 0
>>>>>>> im = 3 ; time = Sat Aug 3 15:40:14 2002
Jan> hybridDesignOpt() called with: weight.margin= 0 , KG.margin= 2.548
Jan> GA Design: ShipMathModel( LWL=578.21472911227, B=49.466359186738,
Jan> Cp=0.702282574662
Jan> H.DKd=11.5397975180451 ) = (Cost=3.25 , Err.Pwr=0.8, Err.Range=-1.5,
Jan> B/H=3.28)
Jan> Garbage collection 636 = 358+130+148 (level 2) ...
Jan> 4432 cons cells free (0%)
Jan> 6.4 Mbytes of heap free (63%)
Jan> used (Mb) gc trigger (Mb) limit (Mb)
Jan> Ncells 560568 15.0 565000 15.1 16.1
Jan> Vcells 472355 3.7 1310720 10.0 20.0
Jan> total used free shared buffers cached
Jan> Mem: 124948 94624 30324 0 17308 25824
Jan> -/+ buffers/cache: 51492 73456
Jan> Swap: 265032 8 265024
Jan> [1] 0
Jan> Error: cons memory exhausted (limit reached?)
Jan> In addition: There were 24 warnings (use warnings() to see them)
Jan> Lost warning messages
>> gc()
Jan> used (Mb) gc trigger (Mb) limit (Mb)
Jan> Ncells 563691 15.1 565000 15.1 16.1
Jan> Vcells 476605 3.7 1310720 10.0 20.0
>> rm(list = ls(all = TRUE))
>> gc()
Jan> used (Mb) gc trigger (Mb) limit (Mb)
Jan> Ncells 554338 14.9 565000 15.1 16.1
Jan> Vcells 467858 3.6 1310720 10.0 20.0
>> rm(list = ls(all = TRUE))
>> gc()
Jan> used (Mb) gc trigger (Mb) limit (Mb)
Jan> Ncells 554338 14.9 565000 15.1 16.1
Jan> Vcells 467858 3.6 1310720 10.0 20.0
>> memory.profile()
Jan> NILSXP SYMSXP LISTSXP CLOSXP ENVSXP PROMSXP
Jan> LANGSXP
Jan> 1 5031 264110 1401 16047 0
Jan> 60670
Jan> SPECIALSXP BUILTINSXP CHARSXP LGLSXP INTSXP
Jan> 59 503 29179 1682 0 0
Jan> 24
Jan> REALSXP CPLXSXP STRSXP DOTSXP ANYSXP VECSXP
Jan> EXPRSXP
Jan> 135616 8 23982 0 0 16053
Jan> 0
Jan> EXTPTRSXP WEAKREFSXP
Jan> 0 0 0
-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.- r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html Send "info", "help", or "[un]subscribe" (in the "body", not the subject !) To: r-help-request at stat.math.ethz.ch _._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._