Manage huge database
What are you going to do with the data once you have read it in? Are all the data items numeric? If they are numeric, you would need at least 8GB to hold one copy and probably a machine with 32GB if you wanted to do any manipulation on the data. You can use a 'connection' and 'scan' to read the data in chunks and then store it in a more accessible format. A lot would depend on your answer to my first question.
On Mon, Sep 22, 2008 at 6:26 AM, Jos? E. Lozano <lozalojo at jcyl.es> wrote:
Maybe you've not lurked on R-help for long enough :) Apologies!
Probably.
So, how much "design" is in this data? If none, and what you've basically got is a 2000x500000 grid of numbers, then maybe a more raw
Exactly, raw data, but a little more complex since all the 500000 variables are in text format, so the width is around 2,500,000.
http://cran.r-project.org/web/packages/RNetCDF/index.html http://cran.r-project.org/web/packages/hdf5/index.html
Thanks, I will check. Right now I am reading line by line the file. It's time consuming, but since I will do it only once, just to rearrange the data into smaller tables to query, it's ok.
Thinking back to your 4GB file with 1,000,000,000 entries, that's only 3 bytes per entry (+1 for the comma). What is this data? There may be more efficient ways to handle it.
Is genetic DNA data (individuals genotyped), hence the large amount of columns to analyze. Best Regards, Jose Lozano ------------------------------------------ Jose E. Lozano Alonso Observatorio de Salud P?blica. Direccion General de Salud P?blica e I+D+I. Junta de Castilla y Le?n. Direccion: Paseo de Zorrilla, n?1. Despacho 3103. CP 47071. Valladolid.
______________________________________________ R-help at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
-- Jim Holtman Cincinnati, OH +1 513 646 9390 What is the problem that you are trying to solve?