Preprocessing RData file (Was: Kdb (Was: high frequency data analysis in R))
Is there any literature on the relative performance gain of preprocessing data into RData and then reading into R? Does it breakdown anywhere? I have 4 GB of data that I'm reading in and I/O is a large bottleneck. Brian -----Original Message----- From: r-sig-finance-bounces at stat.math.ethz.ch [mailto:r-sig-finance-bounces at stat.math.ethz.ch] On Behalf Of Dirk Eddelbuettel Sent: Thursday, May 21, 2009 1:42 PM To: Hae Kyung Im Cc: r-sig-finance at stat.math.ethz.ch Subject: [R-SIG-Finance] Kdb (Was: high frequency data analysis in R)
On 21 May 2009 at 11:13, Hae Kyung Im wrote:
| access (query) this huge database. I looked a little bit into kdb but | you have to pay ~25K to buy the software for one processor. I haven't True, but you can have "free" (as in beer) 32bit version that times out after two hours. That's not a bad compromise. I looked at it for a bit, and I has an R interface. (My blog has a patch to fix their then-broken interface to R's Datetime; I think they may have integrated that by now). Then again you can also pre-process into RData files, or use hdf5, or use a gazillion other methods. But the free trial version may just help for the odd research project like the one Haky described. Dirk
Three out of two people have difficulties with fractions. _______________________________________________ R-SIG-Finance at stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-finance -- Subscriber-posting only. -- If you want to post, subscribe first. -------------------------------------------------------------------------- This message w/attachments (message) may be privileged, confidential or proprietary, and if you are not an intended recipient, please notify the sender, do not use or share it and delete it. Unless specifically indicated, this message is not an offer to sell or a solicitation of any investment products or other financial product or service, an official confirmation of any transaction, or an official statement of Merrill Lynch. Subject to applicable law, Merrill Lynch may monitor, review and retain e-communications (EC) traveling through its networks/systems. The laws of the country of each sender/recipient may impact the handling of EC, and EC may be archived, supervised and produced in countries other than the country in which you are located. This message cannot be guaranteed to be secure or error-free. References to "Merrill Lynch" are references to any company in the Merrill Lynch & Co., Inc. group of companies, which are wholly-owned by Bank of America Corporation. Securities and Insurance Products: * Are Not FDIC Insured * Are Not Bank Guaranteed * May Lose Value * Are Not a Bank Deposit * Are Not a Condition to Any Banking Service or Activity * Are Not Insured by Any Federal Government Agency. Attachments that are part of this E-communication may have additional important disclosures and disclaimers, which you should read. This message is subject to terms available at the following link: http://www.ml.com/e-communications_terms/. By messaging with Merrill Lynch you consent to the foregoing. --------------------------------------------------------------------------