This is a report on my testing with lazyData turned on and off wrt. installation time and memory requirements. It turns out that using lazyData dramatically increases memory consumption and time for a (admittedly large) annotation package. Perhaps this is something we should think about wrt. annotation and data packages. Test example is IlluminaHumanMethylationEPICanno.ilm10b2.hg19 an annotation package for minfi. The .tar.gz for the this package is 113 so its not small. I have explored using LazyData: yes/no in DESCRIPTION adding a single line data/datalist file containing the objects in the package What follows are timings and memory consumption of R CMD build + INSTALL on my Mac laptop using an SSD drive. LazyData: yes datalist: no 285 seconds 3.22 GB (values as high as 3.8GB seen) LazyData: no datalist: no 81s 1.64 GB LazyData: no datalist: yes 19s 0.38 GB (following combination is not mentioned by R-exts, and while it still uses tons of memory, it seems to be 1 minute faster; redid measuring once to confirm this) LazyData: yes datalist: yes 226 s 3.26 GB (values as high as 3.9GB seen) Make the data LazyLoaded is pretty nice; one thing is it avoids polluting the global environment. But it seems that it would be worthwhile to consider if some of this could be done prior to the package build time. Perhaps not, but for sure we are spending resources on the building and installing of this by the build system. I started going down this route because my Travis build starting being killed due to 3+GB being used. I really don't like turning off LazyLoad because of the global environment issue, but the number are kind of extreme here. Best, Kasper
[Bioc-devel] lazyData
3 messages · Kasper Daniel Hansen, Martin Morgan
11 days later
On 07/18/2016 10:52 AM, Kasper Daniel Hansen wrote:
This is a report on my testing with lazyData turned on and off wrt. installation time and memory requirements. It turns out that using lazyData dramatically increases memory consumption and time for a (admittedly large) annotation package. Perhaps this is something we should think about wrt. annotation and data packages. Test example is IlluminaHumanMethylationEPICanno.ilm10b2.hg19 an annotation package for minfi. The .tar.gz for the this package is 113 so its not small. I have explored using LazyData: yes/no in DESCRIPTION adding a single line data/datalist file containing the objects in the package What follows are timings and memory consumption of R CMD build + INSTALL on my Mac laptop using an SSD drive. LazyData: yes datalist: no 285 seconds 3.22 GB (values as high as 3.8GB seen) LazyData: no datalist: no 81s 1.64 GB LazyData: no datalist: yes 19s 0.38 GB
Hi Kasper -- I have to admit my ignorance on the miracle of lazy data. Can you clarify what one gains from LazyData? I kind of though that with LazyData: true the data was only loaded when needed, but that doesn't seem consistent with the picture you paint above? Also, what's the discussion about global variables? Martin
(following combination is not mentioned by R-exts, and while it still uses tons of memory, it seems to be 1 minute faster; redid measuring once to confirm this) LazyData: yes datalist: yes 226 s 3.26 GB (values as high as 3.9GB seen) Make the data LazyLoaded is pretty nice; one thing is it avoids polluting the global environment. But it seems that it would be worthwhile to consider if some of this could be done prior to the package build time. Perhaps not, but for sure we are spending resources on the building and installing of this by the build system. I started going down this route because my Travis build starting being killed due to 3+GB being used. I really don't like turning off LazyLoad because of the global environment issue, but the number are kind of extreme here. Best, Kasper [[alternative HTML version deleted]]
_______________________________________________ Bioc-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel
This email message may contain legally privileged and/or...{{dropped:2}}
With LazyData true you indeed don't load the data until it is available. My guess, from skimming the code extremely fast, is that the extreme requirements (memory and time) during installation is because the data objects needs to get loaded and somehow modified for this to happen. Re. the global environment: if my package has an object TEST, and LazyData is TRUE, when I do (say) data(TEST) or use TEST somehow, TEST doesn't exists in the Global environment. But if LazyData is FALSE and I do data(TEST), TEST gets copied into the Global environment, which is kind of irritating when it is annotation data because it seems fragile to me (perhaps it is not). Best, Kasper On Fri, Jul 29, 2016 at 3:38 PM, Martin Morgan <
martin.morgan at roswellpark.org> wrote:
On 07/18/2016 10:52 AM, Kasper Daniel Hansen wrote:
This is a report on my testing with lazyData turned on and off wrt. installation time and memory requirements. It turns out that using lazyData dramatically increases memory consumption and time for a (admittedly large) annotation package. Perhaps this is something we should think about wrt. annotation and data packages. Test example is IlluminaHumanMethylationEPICanno.ilm10b2.hg19 an annotation package for minfi. The .tar.gz for the this package is 113 so its not small. I have explored using LazyData: yes/no in DESCRIPTION adding a single line data/datalist file containing the objects in the package What follows are timings and memory consumption of R CMD build + INSTALL on my Mac laptop using an SSD drive. LazyData: yes datalist: no 285 seconds 3.22 GB (values as high as 3.8GB seen) LazyData: no datalist: no 81s 1.64 GB LazyData: no datalist: yes 19s 0.38 GB
Hi Kasper -- I have to admit my ignorance on the miracle of lazy data. Can you clarify what one gains from LazyData? I kind of though that with LazyData: true the data was only loaded when needed, but that doesn't seem consistent with the picture you paint above? Also, what's the discussion about global variables? Martin
(following combination is not mentioned by R-exts, and while it still uses
tons of memory, it seems to be 1 minute faster; redid measuring once to
confirm this)
LazyData: yes
datalist: yes
226 s
3.26 GB (values as high as 3.9GB seen)
Make the data LazyLoaded is pretty nice; one thing is it avoids polluting
the global environment.
But it seems that it would be worthwhile to consider if some of this could
be done prior to the package build time. Perhaps not, but for sure we are
spending resources on the building and installing of this by the build
system.
I started going down this route because my Travis build starting being
killed due to 3+GB being used. I really don't like turning off LazyLoad
because of the global environment issue, but the number are kind of
extreme
here.
Best,
Kasper
[[alternative HTML version deleted]]
_______________________________________________ Bioc-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel
This email message may contain legally privileged and/or confidential information. If you are not the intended recipient(s), or the employee or agent responsible for the delivery of this message to the intended recipient(s), you are hereby notified that any disclosure, copying, distribution, or use of this email message is prohibited. If you have received this message in error, please notify the sender immediately by e-mail and delete this email message from your computer. Thank you.