xts + indexing for tick data?
On Thu, Aug 05, 2010 at 02:09:45PM -0500, Jeff Ryan wrote:
The general design is something like a data.frame, but implemented as an environment. The internal structure shouldn't be a concern though, just note that the interface attempts to behave like a data.frame.
Is this interface available in the indexing package, or will it be?
In essence, you can take any type of data that mmap can use and put it into this indexed environment/db.
Many many GB or bigger is easy even on a laptop.
Cool. Is there an example somewhere of how I should take, say, a simple data frame, design an appropriate set of column files and indexes, and then populate them with tons of data? Should I just go ahead and design my own little layout scheme, or would I be needlessly reinventing the wheel?
Nice option contracts use case. For populating your 12 GB of binary data files, did you: 1. Do it with R. 2. Use some of the C code in your mmap or indexing packages to help you, but without R. 3. Do it using some other entirely unrelated code or tool. I'm curious about which approach you found most useful for data loading and maintenance, particularly while making sure that the various different files for each of your 13 columns are kept in sync. I've got a similar amount of futures tick data, currently in a totally unwieldy vendor format, and need to decide how to arrange it...
Andrew Piskorski <atp at piskorski.com> http://www.piskorski.com/