Keeping persistent data collections

Hello, I recently found this list and have been reading deeply the
archives. I am wondering how people here maintain their collections of data
for easy use in R. I am wondering a few things:

1) How do members of this list deal with keeping persistent data
collections with R? I was thinking of individual xts objects by asset and
frequency (such as AAPL daily, AAPL minute, AAPL 60m, etc). While I can
store and maintain these xts objects on disk and load them into R as
needed, I am wondering if there is a more better solution.

2) Coming from that, I have been looking into the indexing package for my
needs. It seems very useful for managing a lot of large data sets in
memory, but I am not sure it is a good method for maintaining persistent
data, I have found trouble adding information to existing data that is
indexed on disk. Do poster here use indexing for this purpose? I did find
an old post or two touching on that with no specifics. I would like to be
able to combine the ability of indexing to have many large data sets
available in memory with persistent storage of data. Has anyone any
experience doing this?

3) How do people keep track of all the data sets within R? Are there any
useful packages for keeping track of multiple sets of financial data and
the information about them?

4) Any other pointers? I know many here are well versed and manage large
data sets with R. Any tips you have or even simply showing me in a helpful
direction to useful packages you use is great. This list is a great help
for me and I am still browsing old threads!

library(FinancialInstrument)

# object with daily periodicity
data(sample_matrix)
DDD <- as.xts(sample_matrix)

#object with minute periodicity
AAA <- xts(rnorm(1:10000), Sys.time()-(60*1:10000))
AAA <- align.time(AAA)
colnames(AAA) <- "AAA"

# look at the objects we're going to store
head(AAA)

head(DDD)

mydir <- getwd()
saveSymbols.days("AAA", base_dir=mydir)
saveSymbols.common("DDD", base_dir=mydir)

# now that they are on disk,
# remove them from workspace
rm("AAA", "DDD")

# get from disk
getSymbols("AAA", src='FI', dir=mydir, split_method='days',

getSymbols("DDD", src='FI', dir=mydir, split_method='common')

head(AAA)

head(DDD)

#--------
# You can setSymbolLookup so that getSymbols will know where
# to look.  There are 2 ways to setSymbolLookup: explicitly,
# or by setting the "src" field of an instrument.

# explicitly
setSymbolLookup(DDD=list(src='FI', dir=mydir, split_method='common'))

getSymbols("DDD")

# by using the "src" field of an instrument
stock("AAA", currency("USD"), src=list(src='FI', dir=mydir,

getSymbols("AAA", from='2011-10-31')

# cleanup
rm("AAA", "DDD")
unlink("AAA", recursive=TRUE)
unlink("DDD", recursive=TRUE)

On Sun, 2011-11-06 at 22:43 -0500, Dino Veritas wrote:

Hello, I recently found this list and have been reading deeply the
archives. I am wondering how people here maintain their collections of

data

for easy use in R. I am wondering a few things:

1) How do members of this list deal with keeping persistent data
collections with R? I was thinking of individual xts objects by asset and
frequency (such as AAPL daily, AAPL minute, AAPL 60m, etc). While I can
store and maintain these xts objects on disk and load them into R as
needed, I am wondering if there is a more better solution.

I store only tick data, as I can easily get to any other frequency from
tick.  I've considered also storing daily data, but in the end I decide
it is too much trouble to (additionally) manage, and just store tick.

2) Coming from that, I have been looking into the indexing package for my
needs. It seems very useful for managing a lot of large data sets in
memory, but I am not sure it is a good method for maintaining persistent
data, I have found trouble adding information to existing data that is
indexed on disk. Do poster here use indexing for this purpose? I did find
an old post or two touching on that with no specifics. I would like to be
able to combine the ability of indexing to have many large data sets
available in memory with persistent storage of data. Has anyone any
experience doing this?

You are correct that the 'indexing' package is very powerful.  It is
also not done yet.

As I said, I store tick data.  The way I do this is with single files
per day of data per symbol, pre-parsed into xts objects and stored to
disk in one directory per symbol (using 'save').

I then use FinancialInstrument to keep track of all the instrument
metadata, and getSymbols to load the data into R when I need it (and
over the time-frames that I require).  We currently download tick data
for about 2500 tradeable instruments per day, and maintain archives
going back several years.  We have the .instrument environment stored on
the same file server as the data, and every .Rprofile in the firm points
to this so that everyone has access to getInstrument and getSymbols

I know someone who works in the hedge fund industry, mostly with monthly
data, with some daily data sprinkled in.  He uses the same approach I
have outlined of storing the metadata in FinancialInstrument, and
getSymbols to access the data.  He typically stores one consolidated CSV
file per instrument, because CSV files are easy to add on to with a
batch process.

For lower frequency data (let's say daily or lower) a database is
certainly an option, and there are getSymbols wrappers that could be
adapted to whatever schema you decided to use. Obviously, there are tick
data database providers such as OneTick and kdb, and if you have this
problem and the resources to need this type of solution, you probably
already know that you are in this camp, and know that these providers
have R interfaces of varying quality.

The FinancialInstrument package has a 'parsers' directory included in
the 'inst' directory of the package with many examples of download and
parse routines for regular loading of data from a variety of free or
subscription providers.  This should give you a lot of material to begin
working with your own data providers.

3) How do people keep track of all the data sets within R? Are there any
useful packages for keeping track of multiple sets of financial data and
the information about them?

We wrote and use FinancialInstrument for this purpose.

As I said earlier, I see no value in storing different periodicities,
and store only tick.

One of the reasons that I chose to write a getSymbols wrapper for
retrieving our tick data stores is that resources like this list have
extensive experience about using getSymbols, and it is therefore easy
for people at our firm to become familiar with using the data.

Also, I am reasonably confident that as the indexing package matures,
there will be a getSymbols method for it as well, and if appropriate I
can easily convert all my data in one batch pass and it will be
transparent to my users.

I made what I now realize to have been a mistake at a previous firm in
writing a data retrieval function that was not compatible with
getSymbols which was more complex to teach people how to use it, and
less compatible with huge amounts of other publicly available code.

quantmod and FinancialInstrument contain examples of various getSymbols
methods that may meet your needs, or that could serve as templates for
your custom in-house data source.

4) Any other pointers? I know many here are well versed and manage large
data sets with R. Any tips you have or even simply showing me in a

helpful

direction to useful packages you use is great. This list is a great help
for me and I am still browsing old threads!

Regards,

   - Brian

--
Brian G. Peterson
http://braverock.com/brian/
Ph: 773-459-4973
IM: bgpbraverock