Preprocessing RData file (data.table and ff, bigmemory)

You could also have a look at filehash, with which I've been playing around for a while. 

Furthermore, I recently found a package still in Beta version, colbycol, written by Carlos J. Gil Bellosta (http://www.datanalytics.com) which seems to do what's on the tin: reading and managing large datasets well beyond the in-process memory limits...

Jos?

-----Original Message-----
From: r-sig-finance-bounces at stat.math.ethz.ch [mailto:r-sig-finance-bounces at stat.math.ethz.ch] On Behalf Of Steve Jaffe
Sent: 22 May 2009 15:19
To: r-sig-finance at stat.math.ethz.ch
Subject: Re: [R-SIG-Finance] [R-sig-finance] Preprocessing RData file (data.table and ff, bigmemory)


I'm new to R and interested in working with large amounts of data
(timeseries, but regularly spaced.) Can you point me to a good reference for
using data.table with bigmemory or ff? 

(I'm a bit puzzled about what exactly these packages provide. As I
understand it, on 32-bit platforms files are subject to the same 2GB limit
as in-process memory, so I assume that dealing with a larger dataset still
requires breaking it up into multiple files...)

Thanks for your help.



I failed to point out that data.table can make use of both (?) those
packages. [ff, bigmemory]

It isn't a time-series library per se, but it make one very cool
in-memory database.  Similar in spirit to some of the not-so-free ones
out there...

Jeff

View this message in context: http://www.nabble.com/high-frequency-data-analysis-in-R-tp23654793p23671313.html
Sent from the Rmetrics mailing list archive at Nabble.com.

_______________________________________________
R-SIG-Finance at stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-finance
-- Subscriber-posting only.
-- If you want to post, subscribe first.