Message-ID: <CAAmySGNrdPexHCTJ3Ro4yxPaXwuyrenHfX2LaiWEzwDr0EUunQ@mail.gmail.com>
Date: 2011-10-03T20:11:38Z
From: R. Michael Weylandt
Subject: file input with readLines
In-Reply-To: <39B5ED61E7BFC24FA8277B6DE92A9A3F041541B6@fkimlki01.enterprise.afmc.ds.af.mil>
If you are using rbind() at each iteration, that can slow things down
greatly. Look up a document called the R Inferno which discusses this
in great detail in circle 2.
Michael Weylandt
On Mon, Oct 3, 2011 at 2:26 PM, Cable, Sam B Civ USAF AFMC AFRL/RVBXI
<Sam.Cable at kirtland.af.mil> wrote:
> More on my previous question ...
>
> I have put in timing statements to try to get a better idea of where the
> problem is, like so:
>
> conn<-file('filename','r')
>
> for (chunk in 1:100000) {
> ? print(paste('begin read at',date()))
> ? Lines<-readLines(conn,n=25)
> ? print(paste('begin processing at',date()))
> ?# process "Lines"
> ? print(paste('end loop at',date()))
> }
>
> Every time I go through the loop, all the date() functions return
> *exactly* the same time! ?It *looks like* it runs through each iteration
> very quickly and then takes longer and longer to simply start the next
> iteration. ?I don't believe this. ?I think R must be doing some kind of
> latency trick or something. ?But, anyway, the point is that I was
> assuming the problem was in the I/O, and now I don't know if it's I/O or
> processing. ?Either way, I don't understand it and would really
> appreciate some wisdom from you guys.
>
> Thanks.
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>