Huge data frames?
On Wed, Aug 28, 2002 at 05:38:50AM +0200, Magnus Lie Hetland wrote:
A friend of mine recently mentioned that he had painlessly imported a data file with 8 columns and 500,000 rows into matlab. When I tried the same thing in R (both Unix and Windows variants) I had little success. The Windows version hung for a very long time, until I eventually more or less ran out of virtual memory; I tried to set the proper memory allocations for the Unix version, but it never seemed satisfied :] I used read.table -- should I have used something else? Is it even
try 'scan'
possible to work with this large files? I assume a memory-mapped binary file would have been quite efficient (as opposed to an in-memory parsed text file) -- is something like that even possible in R?
have a look at the package 'rhdf5' (at least at www.bioconductor.org, not sure it's on CRAN). Not exactly what you describe but could be relevant. Regards, L.
-- Magnus Lie Hetland The Anygui Project http://hetland.org http://anygui.org -.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.- r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html Send "info", "help", or "[un]subscribe" (in the "body", not the subject !) To: r-help-request at stat.math.ethz.ch _._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._
-------------------------------------------------------------- Laurent Gautier CBS, Building 208, DTU PhD. Student DK-2800 Lyngby,Denmark tel: +45 45 25 24 89 http://www.cbs.dtu.dk/laurent -.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.- r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html Send "info", "help", or "[un]subscribe" (in the "body", not the subject !) To: r-help-request at stat.math.ethz.ch _._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._