load ing and saving R objects

On Tue, 14 Jun 2005, Prof Brian Ripley wrote:
If your file system does not like 15000 files you can always 
save in a DBMS.

-----Original Message-----
From: Prof Brian Ripley [mailto:ripley at stats.ox.ac.uk]
Sent: Tuesday, June 14, 2005 10:41 AM
To: Barry Rowlingson
Cc: r-help at stat.math.ethz.ch; Richard Mott
Subject: Re: [R] load ing and saving R objects


On Tue, 14 Jun 2005, Barry Rowlingson wrote:

Richard Mott wrote:

Does anyone know a way to do the following:

Save a large number of R objects to a file (like load() 

does) but then

read back only a small named subset of them . As far as I can see,
load() reads back everything.

 Save them to individual files when you generate them?

 for(i in 1:15000){

  m=generateBigMatrix(i)

  filename=paste("BigMatrix-",i,".Rdata",sep='')
  save(m,file=filename)
 }

Note that load will always overwrite 'm', so to load a 

sample of them in

you'll need to do something like this:

 bigSamples=list()

 for(i in sample(15000,N)){
   filename=paste("BigMatrix-",i,".Rdata",sep='')
   load(filename)
   bigSamples[[i]]=m
 }

 But there may be a more efficient way to string up a big list like
that, I can never remember - get it working, then worry 

about optimisation.

(Yes, use bigSamples <- vector("list", 15000) first.)

 I hope your filesystem is happy with 15000 objects in it. I would
dedicate a folder or directory for just these objects' 

files, since it

then becomes near impossible to see anything other than the 

big matrix

files...

.readRDS/.saveRDS might be a better way to do this, and avoids always 
restoring to "m".

If your file system does not like 15000 files you can always 
save in a 
DBMS.

I did once look into restoring just some of the objects in a save()ed 
file, but it is not really possible to do so efficiently due 
to sharing 
between objects.

-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595