How to read large file

3 messages · Dongyan Song, Gabor Grothendieck, Darren Norris

Original

1

3

Dongyan Song

Mon, Mar 16, 2009 8:11 AM #

Dear all,

I want to read a 1.2 G file with 136,736,218 numbers with scan function, and
then want make a histogram with those numbers, but it showed Error: cannot
allocate vector of size 2.0 Gb. How can I import the file then?

The version I used:

R version 2.8.1 (2008-12-22) 
i686-pc-linux-gnu

nsize vsize 
   NA    NA
and the computer I use has 4G memory in total.

Thank you very much! 

Dongyan Song

View this message in context: http://www.nabble.com/How-to-read-large-file-tp22540291p22540291.html
Sent from the R help mailing list archive at Nabble.com.

Gabor Grothendieck

Mon, Mar 16, 2009 8:24 AM #

Example 6e of
http://code.google.com/p/sqldf/#Example_6._File_Input

shows how to use the sqldf package to read a sample of rows
from a file without reading the file itself into R.

On Mon, Mar 16, 2009 at 11:11 AM, Dongyan Song <yzhskdls at hotmail.com> wrote:

______________________________________________
R-help at r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Mon, Mar 16, 2009 1:47 PM #

Try putting "memory limit linux" or "mem.limits" in the nabble help. Or type
?mem.limits in the R gui.
Also have a look at:
http://www.nabble.com/RAM%2C-swap%2C-Error%3A-cannot-allocate-vector-of-size%2C-Linux%3A-tt11680213.html#a11680213

Think you also need to check / understand how R deals with what you want to
do so you know how many copies of the data it will need to hold in memory.

Hope that helps,
Darren

Dongyan Song wrote:

View this message in context: http://www.nabble.com/How-to-read-large-file-tp22540291p22547077.html
Sent from the R help mailing list archive at Nabble.com.