Memory usage in read.csv()
I read vmstat data in just fine without any problems.? Here is an
example of how I do it:
VMstat <- read.table('vmstat.txt', header=TRUE, as.is=TRUE)
vmstat.txt looks like this:
date time r b w swap free re mf pi po fr de sr intr syscalls cs user sys id
07/27/05 00:13:06 0 0 0 27755440 13051648 20 86 0 0 0 0 0 456 2918 1323 0 1 99
07/27/05 00:13:36 0 0 0 27755280 13051480 11 53 0 0 0 0 0 399 1722 1411 0 1 99
07/27/05 00:14:06 0 0 0 27753952 13051248 18 88 0 0 0 0 0 424 1259 1254 0 1 99
07/27/05 00:14:36 0 0 0 27755304 13051496 17 85 0 0 0 0 0 430 1029 1246 0 1 99
07/27/05 00:15:06 0 0 0 27755064 13051232 41 278 0 1 1 0 0 452 2047 1386 0 1 99
07/27/05 00:15:36 0 0 0 27753824 13040720 125 1039 0 0 0 0 0 664 4097
1901 3 2 95
07/27/05 00:16:06 0 0 0 27754472 13027000 15 91 0 0 0 0 0 432 1160 1273 0 1 99
07/27/05 00:16:36 0 0 0 27754568 13027104 17 85 0 0 0 0 0 416 1058 1271 0 1 99
Have you tried a smaller portion of data?
Here is what it took to read in a file with 85K lines:
system.time(vmstat <- read.table('c:/vmstat.txt', header=TRUE))
user system elapsed 2.01 0.01 2.03
str(vmstat)
'data.frame': 85680 obs. of 20 variables: $ date : Factor w/ 2 levels "07/27/05","07/28/05": 1 1 1 1 1 1 1 1 1 1 ... $ time : Factor w/ 2856 levels "00:00:26","00:00:56",..: 27 29 31 33 35 37 39 41 43 45 ... $ r : int 0 0 0 0 0 0 0 0 0 0 ... $ b : int 0 0 0 0 0 0 0 0 0 0 ... $ w : int 0 0 0 0 0 0 0 0 0 0 ... $ swap : int 27755440 27755280 27753952 27755304 27755064 27753824 27754472 27754568 27754560 27754704 ... $ free : int 13051648 13051480 13051248 13051496 13051232 13040720 13027000 13027104 13027096 13027240 ... $ re : int 20 11 18 17 41 125 15 17 13 12 ... $ mf : int 86 53 88 85 278 1039 91 85 69 51 ... $ pi : int 0 0 0 0 0 0 0 0 0 0 ... $ po : int 0 0 0 0 1 0 0 0 0 1 ... $ fr : int 0 0 0 0 1 0 0 0 0 1 ... $ de : int 0 0 0 0 0 0 0 0 0 0 ... $ sr : int 0 0 0 0 0 0 0 0 0 0 ... $ intr : int 456 399 424 430 452 664 432 416 425 432 ... $ syscalls: int 2918 1722 1259 1029 2047 4097 1160 1058 1198 1727 ... $ cs : int 1323 1411 1254 1246 1386 1901 1273 1271 1268 1477 ... $ user : int 0 0 0 0 0 3 0 0 0 0 ... $ sys : int 1 1 1 1 1 2 1 1 1 1 ... $ id : int 99 99 99 99 99 95 99 99 99 99 ...
On Tue, Jan 19, 2010 at 9:25 AM, <nabble.30.miller_2555 at spamgourmet.com> wrote:
I'm sure this has gotten some attention before, but I have two CSV
files generated from vmstat and free that are roughly 6-8 Mb (about
80,000 lines) each. When I try to use read.csv(), R allocates all
available memory (about 4.9 Gb) when loading the files, which is over
300 times the size of the raw data. ?Here are the scripts used to
generate the CSV files as well as the R code:
Scripts (run for roughly a 24-hour period):
? ?vmstat -ant 1 | awk '$0 !~ /(proc|free)/ {FS=" "; OFS=","; print
strftime("%F %T %Z"),$6,$7,$12,$13,$14,$15,$16,$17;}' >>
~/vmstat_20100118_133845.o;
? ?free -ms 1 | awk '$0 ~ /Mem\:/ {FS=" "; OFS=","; print
strftime("%F %T %Z"),$2,$3,$4,$5,$6,$7}' >>
~/memfree_20100118_140845.o;
R code:
? ?infile.vms <- "~/vmstat_20100118_133845.o";
? ?infile.mem <- "~/memfree_20100118_140845.o";
? ?vms.colnames <-
c("time","r","b","swpd","free","inact","active","si","so","bi","bo","in","cs","us","sy","id","wa","st");
? ?vms.colclass <- c("character",rep("integer",length(vms.colnames)-1));
? ?mem.colnames <- c("time","total","used","free","shared","buffers","cached");
? ?mem.colclass <- c("character",rep("integer",length(mem.colnames)-1));
? ?vmsdf <- (read.csv(infile.vms,header=FALSE,colClasses=vms.colclass,col.names=vms.colnames));
? ?memdf <- (read.csv(infile.mem,header=FALSE,colClasses=mem.colclass,col.names=mem.colnames));
I am running R v2.10.0 on a 64-bit machine with Fedora 10 (Linux
version 2.6.27.41-170.2.117.fc10.x86_64 ) with 6Gb of memory. There
are no other significant programs running and `rm()` followed by `
gc()` successfully frees the memory (followed by swapins after other
programs seek to used previously cached information swapped to disk).
I've incorporated the memory-saving suggestions in the `read.csv()`
manual page, excluding the limit on the lines read (which shouldn't
really be necessary here since we're only talking about < 20 Mb of raw
data. Any suggestions, or is the read.csv() code known to have memory
leak/ overcommit issues?
Thanks
______________________________________________ R-help at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
-- Jim Holtman Cincinnati, OH +1 513 646 9390 What is the problem that you are trying to solve?