Skip to content

Reading and coalescing many datafiles.

3 messages · asr@ufl.edu, Roger D. Peng, Peter Dalgaard

asr@ufl.edu
# 
Greetings.


I've got some analysis problems I'm trying to solve, the raw data for which
are accumulated in a bunch of time-and-date-based files.

/some/path/2005-01-02-00-00-02

etc.


The best 'read all these files' method I've seen in the r-help archives comes
down to 

for (df in my_list_of_filenames )
    {
          dat <- rbind(dat,my_read_function(df))
    } 

which, unpleasantly, is O(N^2) w.r.t. the number of files.

I'm fiddling with other idioms to accomplish the same goal.  Best I've come up
with so far, after extensive reference to the mailing list archives, is


my_read_function.many<-function(filenames)
  {
    filenames <- filenames[file.exists(filenames)];
    rv <- do.call("rbind", lapply(filenames,my_read_function))
    row.names(rv) = c(1:length(row.names(rv)))
    rv
  }


I'd love to have some stupid omission pointed out.


- Allen S. Rout
Roger D. Peng
# 
In my experience, using 'do.call("rbind", ...)' after storing all the 
data files in a list is much better than 'rbind'-ing on the fly.

-roger
asr at ufl.edu wrote:

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Peter Dalgaard
    

    

      
      #
    

  


  

      
asr at ufl.edu writes:

            

      
      
      
    

        
Why? It's pretty much what I would suggest, except for the superfluous
c().