On 12-12-20 6:26 PM, Peter Meissner wrote:
Hey,
I have an double loop like this:
chunk <- list(1:10, 11:20, 21:30)
for(k in 1:length(chunk)){
print(chunk[k])
DummyCatcher <- NULL
for(i in chunk[k]){
print("i load something")
dummy <- 1
print("i do something")
dummy <- dummy + 1
print("i do put it together")
DummyCatcher = rbind(DummyCatcher, dummy)
}
print("i save a chunk and restart with another chunk of data")
}
The problem now is that with each 'chunk'-cycle the memory used by R
becomes bigger and bigger until it exceeds my RAM but the RAM it needs
for any of the chunk-cycles alone is only a 1/5th of what I have overall.
Does somebody have an idea why this behaviour might occur? Note that all
the objects (like 'DummyCatcher') are reused every cycle so that I would
assume that the RAM used should stay about the same after the first
'chunk' cycle.
You should pre-allocate your result matrix. By growing it a few rows at
a time, R needs to do this:
allocate it
allocate a bigger one, copy the old one in
delete the old one, leaving a small hole in memory
allocate a bigger one, copy the old one in
delete the old one, leaving a bigger hold in memory, but still too small
to use...
etc.
If you are lucky, R might be able to combine some of those small holes
into a bigger one and use that, but chances are other variables will
have been created there in the meantime, so the holes will go mostly
unused. R never moves an object during garbage collection, so if you
have fragmented memory, it's mostly wasted.
If you don't know how big the final result will be, then allocate large,
and when you run out, allocate bigger. Not as good as one allocation,
but better than hundreds.
Duncan Murdoch
Best, Peter
SystemInfo:
R version 2.15.2 (2012-10-26)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Win7 Enterprise, 8 GB RAM