Process File Line By Line Without Slurping into Object
Gundala Viswanath wrote:
Dear all,
In general practice one would slurp the whole file
using this method before processing the data:
dat <- read.table(filename)
or variations of it.
Is there a way we can access the file line by line
without slurping/storing them into object?
I am thinking something like this in Perl:
__BEGIN__
open INFILE, '<' , 'filename.txt' or die $!;
while (<INFILE>) {
my $line = $_;
# then process line by line
}
__END__
the reason I want to do that is because the data
I am processing are large (~ 5GB), my PC may not
be able to handle that.
Use a connection. For example,
con <- file("filename.txt", "rt")
repeat {
line <- readLines(con, 1)
if (!length(line)) break
# process the line, or change the "1" to a bigger number, and
process the block of lines
}
close(con)
Duncan Murdoch