good and bad ways to import fixed column data (rpy)

Recorded here so others may avoid my mistakes.

I have a bunch of files containing fixed width data. ?The R Data guide
suggests that one pre-process them with a script if they are large.
They were 50MG and up, and I needed to process another file that gave
the layout of the lines anyway.

I tried rpy to not only preprocess but create the R data object in one
go. ?It seemed like a good idea; it wasn't. ?The core operation, was to
build up a string for each line that looked like "data.frame(var1=val1,
var2=val2, [etc])" and then rbind this to the data.frame so far. ?I did
this with r(mycommand string). Almost all the values were numeric.

This was incredibly slow, being unable to complete after running
overnight.

So, the lesson is, don't do that!

I switched to preprocessing that created a csv file, and then read.csv
from R. ?This worked in under a minute. ?The result had dimension 150913
x 129.

The good news in rpy was that I found objects persisted across calls to
the r object.

Exactly why this was so slow I don't know. ?The two obvious suspects the
speed of rbind, which I think is pretty inefficient, and the overhead of
crossing the python/R boundary.

This was on Debian Lenny:
python-rpy ? ? ? ? ? ? ? ? ? ?1.0.3-2
Python 2.5.2
R 2.7.1

rpy2 is not available in Lenny, though it is in development versions of
Debian.

Ross Boylan

Gabor made a good point.
Here is an example I copied from my blog.
 
##############################################
# READ FIXED-WIDTH DATA FILE WITH read.fwf() #
# ------------------------------------------ #
# EQUIVALENT SAS CODE:                       #
# filename data 'E:\sas\fixed.txt';          #
# data test;                                 #
#   infile data truncover;                   #
#   input @1 city $ 1 - 22 @23 population;   #
# run;                                       #
##############################################

# OPEN A CONNECTION TO THE DATA FILE
data <- file(description = "e:\\sas\\fixed.txt", open = "r")

# width = c(...)      ==> SPECIFIES COLUMN WIDTHS
# col.names = c(...)  ==> GIVES COLUMN NAMES
# colClasses = c(...) ==> DEFINES COLUMN CLASSES
test <- read.fwf(data, header = FALSE, width = c(22, 10), 
                 col.names = c("city", "population"),
                 colClasses = c("character", "numeric"))

close(data)
 
On Sun, Aug 16, 2009 at 6:36 PM, Gabor
Grothendieck<ggrothendieck at gmail.com> wrote:

Check out ?read.fwf

On Sun, Aug 16, 2009 at 4:49 PM, Ross Boylan<ross at biostat.ucsf.edu>

wrote:

Recorded here so others may avoid my mistakes.

I have a bunch of files containing fixed width data.  The R Data

guide

suggests that one pre-process them with a script if they are large.
They were 50MG and up, and I needed to process another file that

gave

the layout of the lines anyway.

I tried rpy to not only preprocess but create the R data object in

one

go.  It seemed like a good idea; it wasn't.  The core operation,

was to

build up a string for each line that looked like

"data.frame(var1=val1,

var2=val2, [etc])" and then rbind this to the data.frame so far.  I

did

this with r(mycommand string). Almost all the values were numeric.

This was incredibly slow, being unable to complete after running
overnight.

So, the lesson is, don't do that!

I switched to preprocessing that created a csv file, and then

read.csv

from R.  This worked in under a minute.  The result had dimension

150913

x 129.

The good news in rpy was that I found objects persisted across

calls to

the r object.

Exactly why this was so slow I don't know.  The two obvious

suspects the

speed of rbind, which I think is pretty inefficient, and the

overhead of

crossing the python/R boundary.

This was on Debian Lenny:
python-rpy                    1.0.3-2
Python 2.5.2
R 2.7.1

rpy2 is not available in Lenny, though it is in development

versions of

Debian.

Ross Boylan