An embedded and charset-unspecified text was scrubbed... Name: not available URL: <https://stat.ethz.ch/pipermail/r-help/attachments/20090319/c0293ca9/attachment-0002.pl>
Reading a file line by line - separating lines VS separating columns
2 messages · Tal Galili, jim holtman
You can do something like this using connections and read in a set of
lines and saving the results in bigmemory, or in this case a 'save'
image:
zz <- file("ex.data", "w") # open an output file
for (i in 1:10000)cat( "1\t2\t3\t4\t5\t6\t7\t8\t9\t10\t\t555\t\t",
file = zz, sep ="\n")
close(zz)
# read in the data 876 lines at a time and write out an image
zz <- file("ex.data", "r")
fileNo <- 1
repeat{
gotError <- 1 # set to 2 if there is an error
# catch the error if not more data
tryCatch(input <- read.table(zz, nrows=876, sep='\t'),
error=function(x) gotError <<- 2)
if (gotError == 2) break
# save the intermediate data
save(input, file=sprintf("file%03d.RDData", fileNo))
fileNo <- fileNo + 1
}
close(zz)
On Wed, Mar 18, 2009 at 7:17 PM, Tal Galili <tal.galili at gmail.com> wrote:
Hello all.
I wish to read a large data set into R. ?My current issue is in getting the
data so that R would be able to access it. ?Using read.table won't work
since the data is over 1GB in size (and I am using windows XP), so my plan
was to read the file chunk by chunk and each time move it into bigmemory
(I'll play with that when the time will come, maybe ff is better ?!).
I encountered a problem with separating lines VS separating columns, to
which I found a solution but it doesn't feel to be a smart solution, any
ideas or help of how to improve this would be welcomed.
# sample code:
# creating a simple file zz <- file("ex.data", "w") # open an output file
connection cat( "1\t2\t3\t4\t5\t6\t7\t8\t9\t10\t\t555\t\t", file = zz, sep =
"\n") cat( "1\t2\t3\t4\t5\t6\t7\t8\t9\t10\t\t555\t\t", file = zz, sep =
"\n") cat( "1\t2\t3\t4\t5\t6\t7\t8\t9\t10\t\t555\t\t", file = zz, sep =
"\n") (temp.file = scan("ex.data", what = "", sep = "\n")) # here we can
limit the amount of rows we want to use and start from a specific row using
skip # or: #(aa = readLines("ex.data")) str(aa) # we get a vector of
character new.df <- NULL # we go through the vector to split the columns
for(i in 1:length(aa)) { new.df <- rbind(new.df
,unlist(strsplit(temp.file[i], "\t"))) } new.df # or maybe
apply(as.data.frame(temp.file), 1, function(b) unlist(strsplit(b, "\t"))) #
but this transposes the matrix
Thanks,
Tal
--
----------------------------------------------
My contact information:
Tal Galili
Phone number: 972-50-3373767
FaceBook: Tal Galili
My Blogs:
http://www.r-statistics.com/
http://www.talgalili.com
http://www.biostatistics.co.il
? ? ? ?[[alternative HTML version deleted]]
______________________________________________ R-help at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Jim Holtman Cincinnati, OH +1 513 646 9390 What is the problem that you are trying to solve?