large survey data
Douglas Bates <bates at stat.wisc.edu> writes:
Micha? Bojanowski <bojanr at wp.pl> writes:
Recently I came across a problem. I have to analyze a large survey data - something about 600 columns and 10000 rows (tab-delimited file with names in the header). I was able do import the data into an object, but there is no more memory left. Is there a way to import the data column by column? I have to analyze the whole data, but only two variables at a time.
You will probably need to do the data manipulation externally. Two possible solutions are to use a scripting language like python or perl or to store the data in a relational database like PostgreSQL or MySQL. For data of this size I would recommend the relational database approach. R has packages to connect to PostgreSQL or to MySQL.
We didn't see what OS this came from so it might well be Windows.... There, you have some possibilities of setting up an ODBC connection to a text file (via Control Panel - slightly cryptic, but I managed to get it to play at some point). You should be able to access the table as a database using the RODBC package and that will allow you to do the selection of cases/variables.
O__ ---- Peter Dalgaard Blegdamsvej 3 c/ /'_ --- Dept. of Biostatistics 2200 Cph. N (*) \(*) -- University of Copenhagen Denmark Ph: (+45) 35327918 ~~~~~~~~~~ - (p.dalgaard at biostat.ku.dk) FAX: (+45) 35327907 -.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.- r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html Send "info", "help", or "[un]subscribe" (in the "body", not the subject !) To: r-help-request at stat.math.ethz.ch _._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._