read.delim very slow in reading files with lots of columns

system.time(input <- scan("/tempxx.txt", what=0, sep=','))

object.size(input)

On Fri, 25 Sep 2009, Ping-Hsun Hsieh wrote:

Thanks, Ben.

The matrix is a pure numeric matrix (6x700000, 31mb).
I tried the colClasses='numeric' as well as nrows=7(one of these is header
line) on the matrix.
Also I tested it with not setting the two options in read.delim()

A couple of things come to mind.

First, I have not read the internals of scan, but suspect that parsing a
really long line may be slowing things down.

Since you are attempting to read in a numeric matrix, you can simply do a
global replacement of your delimiter with a newline and use scan on the
result. On unix-like systems, something like

? ? ? ?tmp <- scan( pipe( 'tr "\t" "\n" ?< test_data.txt' ) )

ought to help.

Second, the memory occupied by each line - once it has been processed - is
spread over the full 32MB (or 3.2 GB for the 600 by 700000 version) region
of memory. I am guessing that this is causing your cache to work hard to put
it in place.

If you really want the result to be a 600 by 700000 matrix, you might try to
read it in smaller blocks using scan( pipe( "cut ... " ) ) to feed selected
blocks of columns of your text file to R.

HTH,

Chuck

Here is the time spent on reading the matrix for each test.

system.time( tmp <- read.delim("test_data.txt"))

? ?user ? ?system ? elapsed
50985.421 ? ?27.665 51013.384

system.time(tmp <-
read.delim("test_data.txt",colClasses="numeric",nrows=7,comment.char=""))

? ?user ? ?system ? elapsed
51301.563 ? ?60.491 51362.208

It seems setting the options does not speed up the reading at all.
Is it because of the header line? I will test it.
Did I misunderstand something?

One additional and interesting observation:
The one with the options does save memory a lot. It took ~150mb, while the
other took ~4GB for reading the matrix.

I will try the scan() and see if it helps.

Thanks!
Mike


-----Original Message-----
From: Benilton Carvalho [mailto:bcarvalh at jhsph.edu]
Sent: Wednesday, September 23, 2009 4:56 PM
To: Ping-Hsun Hsieh
Cc: r-help at r-project.org
Subject: Re: [R] read.delim very slow in reading files with lots of
columns

use the 'colClasses' argument and you can also set 'nrows'.

b

On Sep 23, 2009, at 8:24 PM, Ping-Hsun Hsieh wrote:

Hi,



I am trying to read a tab-delimited file into R (Ver. 2.8). The
machine I am using is 64bit Linux with 16 GB.

The file is basically a matrix(~600x700000) and as large as 3GB.



The read.delim() ran extremely slow (hours) even with a subset of
the file (31 MB with 6x700000)

I monitored the memory usage, and found it constantly only took less
than 1% of 16GB memory.

Does read.delim() have difficulty to read files with lots of columns?

Any suggestions?



Thanks,

Mike




? ? ? [[alternative HTML version deleted]]

Charles C. Berry ? ? ? ? ? ? ? ? ? ? ? ? ? ?(858) 534-2098
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?Dept of Family/Preventive
Medicine
E mailto:cberry at tajo.ucsd.edu ? ? ? ? ? ? ? UC San Diego
http://famprevmed.ucsd.edu/faculty/cberry/ ?La Jolla, San Diego 92093-0901