speed issues? read R_inferno by Patrick Burns: & a memory query
Actually the issue is not the size of memory that is consumed, but that memory allocation takes place and the object is copied in each iteration of the "bad" loop you have given below. This is not required for the second loop, where R can allocate the memory at once and does not need to copy the object around. Uwe Ligges
On 23.12.2010 14:13, maddox wrote:
Hi,
I'm just starting out with R and came across R_inferno.pdf by Patrick Burns
just yesterday - I recommend it!
His description of how 'growing' objects (e.g. obj<- c(obj,
additionalValue) eats up memory prompted me to rewrite a function (which
made such calls ~210 times) so that it used indexing into a dimensioned
object instead (i.e. obj[i, ]<- additionalValue).
This transformed the process from
old version:
user system elapsed
133.436 14.257 155.807
new version:
user system elapsed
16.041 1.180 18.535
To say I'm delighted is understatement. Thanks for putting the Inferno
together, Patrick.
However I'm misunderstanding the effect this has on memory use, (or
misunderstanding the code I've highjacked to look at memory use). To look at
virtual memory use I'm using the code below from this forum:
cmd<- paste("ps -o vsz", Sys.getpid())
cat("\nVirtual size: ", system(cmd, intern = TRUE) [2], "\n", sep = "")
I did three runs of the old version, and three with the new, preceding each
with gc()& got the outputs below. In summary, the runs of old method
required 17712, 17744& 17744& runs of new method required 13788, 15140&
13656.
Two questions:
1. why does each run of the same process not make the same demand on memory?
They're doing exactly the same work& creating exactly the same new objects.
2. is the modest decrease in memory consumed by new method expected? (having
read R_Inferno I was, perhaps naively, expecting more of an improvement)
? or am I missing something (more than likely! )
Thanks
M
gc()
used (Mb) gc trigger (Mb) max used (Mb) Ncells 786300 21.0 1265230 33.8 1166886 31.2 Vcells 948412 7.3 3244126 24.8 3766604 28.8
cat("old version")
Virtual size before call: 881692
user system elapsed
131.872 14.417 159.653
Virtual size after call: 899404
899404-881692
[1] 17712 ##################
gc()
used (Mb) gc trigger (Mb) max used (Mb) Ncells 786294 21.0 1265230 33.8 1166886 31.2 Vcells 948407 7.3 3244126 24.8 3766604 28.8
cat("old version")
Virtual size before call: 881660
user system elapsed
133.281 14.473 159.661
Virtual size after call: 899440
899440-881660
[1] 17780 ##################
gc()
used (Mb) gc trigger (Mb) max used (Mb) Ncells 786294 21.0 1265230 33.8 1166886 31.2 Vcells 948407 7.3 3244126 24.8 3766604 28.8
cat("old version")
Virtual size before call: 881696
user system elapsed
133.436 14.257 155.807
Virtual size after call: 899440
899440-881696
[1] 17744 ################## ##################
gc()
used (Mb) gc trigger (Mb) max used (Mb) Ncells 786413 21.0 1265230 33.8 1166886 31.2 Vcells 948460 7.3 3244126 24.8 3766604 28.8
cat("new version")
Virtual size before call: 881696
user system elapsed
16.041 1.180 18.535
Virtual size after call: 895484
895484-881696
[1] 13788 ##################
gc()
used (Mb) gc trigger (Mb) max used (Mb) Ncells 786441 21.1 1265230 33.8 1166886 31.2 Vcells 948480 7.3 3244126 24.8 3766604 28.8
cat("new version")
Virtual size before call: 882648
user system elapsed
16.321 1.068 18.136
Virtual size after call: 897788
897788- 882648
[1] 15140 ##################
gc()
used (Mb) gc trigger (Mb) max used (Mb) Ncells 786441 21.1 1265230 33.8 1166886 31.2 Vcells 948480 7.3 3244126 24.8 3766604 28.8
cat("new version")
Virtual size before call: 882648
user system elapsed
16.581 0.992 19.351
Virtual size after call: 896304
896304-882648
[1] 13656