Skip to content

Rapid Random Access

4 messages · Sean Davis, Tony Plate, Barry Rowlingson

Barry Rowlingson
# 
I have some code that can potentially produce a huge number of 
large-ish R data frames, each of a different number of rows. All the 
data frames together will be way too big to keep in R's memory, but 
we'll assume a single one is manageable. It's just when there's a 
million of them that the machine might start to burn up.

  However I might, for example, want to compute some averages over the 
elements in the data frames. Or I might want to sample ten of them at 
random and do some plots. What I need is rapid random access to data 
stored in external files.

  Here's some ideas I've had:

  * Store all the data in an HDF-5 file - problem here is that the 
current HDF package for R reads the whole file in at once.

  * Store the data in some other custom binary format with an index for 
rapid access to the N-th elements. Problems: feels like reinventing HDF, 
cross-platform issues, etc.

  * Store the data in a number of .RData files in a directory. Hence to 
get the N-th element just attach(paste("foo/A-",n,'.RData')) give or 
take a parameter or two.

  * Use a database. Seems a bit heavyweight, but maybe using RSQLite 
could work in order to keep it local.

  What I'm currently doing is keeping it OO enough that I can in theory 
implement all of the above. At the moment I have an implementation that 
does keep them all in R's memory as a list of data frames, which is fine 
for small test cases but things are going to get big shortly. Any other 
ideas or hints are welcome.

thanks

Barry
Tony Plate
#
Barry Rowlingson wrote:
This is exactly the type of situation that the trackObjs package is 
designed for.  It will automatically (and invisibly) store each object 
in its own .RData file so that objects can be accessed as ordinary R 
objects, but are not kept in memory (actually, there are options to 
control whether or not objects are cached in memory). It also caches 
some characteristics of objects so that a brief summary of objects can 
be provided without having to read each object.  The g.data package and 
the filehash package also do similar things wrt to providing automatic 
access to objects in .RData files (and were part of the inspiration for 
the trackObjs package.)

-- Tony Plate
Barry Rowlingson
#
Tony Plate wrote:

            

      
      
      
    

        
Ooh, I'm having deja-vu - yes I think I saw this package a while back 
and wondered what magic it did. I shall go play with it later.

Thanks

Barry