The ratio of object size to rds file size depends on the object. Some
variation is due to how header information is stored in memory and in the
file but I suspect most is due to how compression works (e.g., a vector of
repeated values can be compressed into a smaller file than a bunch of
random bytes).
f <- function (data, ...) {
force(data)
tf <- tempfile()
on.exit(unlink(tf))
save(data, file = tf)
c(`obj/file size` = as.numeric(object.size(data)/file.size(tf)))
}
f(rep(0,1e6), compress=FALSE)
f(sample(as.raw(0:255),size=8e6,replace=TRUE))
f(rep(as.raw(0:255),length=8e6))
obj/file size
23.5567
Bill Dunlap
TIBCO Software
wdunlap tibco.com
On Tue, Sep 18, 2018 at 8:28 AM, Joris Meys <jorismeys at gmail.com> wrote:
Dear all,
I tried to read in a 3.8Gb RDS file on a computer with 16Gb available
memory. To my astonishment, the memory footprint of R rises quickly to
over
13Gb and the attempt ends with an error that says "cannot allocate vector
of size 5.8Gb".
I would expect that 3 times the memory would be enough to read in that
file, but apparently I was wrong. I checked the memory.limit() and that
one
gave me a value of more than 13Gb. So I wondered if this was to be
expected, or if there could be an underlying reason why this file doesn't
want to open.
Thank you in advance
Joris
--
Joris Meys
Statistical consultant
Department of Data Analysis and Mathematical Modelling
Ghent University
Coupure Links 653, B-9000 Gent (Belgium)
<
https://maps.google.com/?q=Coupure+links+653,%C2%A0B-9000+Gent,%C2%A0Belgium&entry=gmail&source=g