the quote problem with readLines()
Hi, Thank you for your concern! The file has 136,047,472 lines, with one value in each line, and is 1.7G in size. I run in a Linux (OpenSuse OS) with 4G memory in total. The error message is Error: cannot allocate vector of size 2.0 Gb. And the worst thing is even if I read all the data into R after I truncate the numbers' precision, i.e. from 1.234567e+00 to 1.2, I cannot manipulate these numbers, for example, I cannot do ks.test, histogram, kernel density estimator, which I want to do with these numbers. And after I input commands above, computer also give error messages like Error: cannot allocate vector of size 809.1 Mb. I can read a half of file, but I want to know the overall distribution of those numbers, and values in this file is not ordered, and it is not quite easy to random pick up some numbers or sort them. Is these information enough? Thank you again! Best, Dongyan
jholtman wrote:
readLines is doing exactly what you are asking: Value A character vector of length the number of lines read. You still have to convert the character strings to numeric. Exactly how large is "quite large"? What system are you running on? How much memory do you have? What is the error message that you are getting? Exactly what does your file look like? Have you tried reading in portions of the file? How big will it be if you could read it in? Will it take up more than 25% of real memory? There is still some information you need to provide so an assessment can be made. On Tue, Mar 17, 2009 at 8:50 AM, Dongyan Song <yzhskdls at hotmail.com> wrote:
Dear all, I read a file with all numbers with readLines function, as below,
f <- file("data.txt")
a <- readLines(f)
but all the values in a are in format "....", and I cannot do the calculation with them since they are not numeric. I wonder how should I skip those quotes, thank you for help! I have to use readLines function instead of scan, read.table or matrix, because the size of file is quite large, and other function cannot allocate enough space/memory to read the input file. Best, Dongyan -- View this message in context: http://www.nabble.com/the-quote-problem-with-readLines%28%29-tp22558454p22558454.html Sent from the R help mailing list archive at Nabble.com.
______________________________________________ R-help at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
-- Jim Holtman Cincinnati, OH +1 513 646 9390 What is the problem that you are trying to solve?
______________________________________________ R-help at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
----- Dongyan Song, Msc Medical informatics, Uppsala University, Sweden
View this message in context: http://www.nabble.com/the-quote-problem-with-readLines%28%29-tp22558454p22579163.html Sent from the R help mailing list archive at Nabble.com.