Memory issues in R
If by "review" you mean read in summary information then sqldf can do that using the sqlite database in two lines of code. You don't have to install, set up or define the database at all. sqldf and the underlying RSQLite will do all that for you. See example 6b on the home page: http://code.google.com/p/sqldf/#Example_6._File_Input On Sun, Apr 26, 2009 at 11:20 AM, Neotropical bat risk assessments
<neotropical.bats at gmail.com> wrote:
? How do people deal with R and memory issues?
? I have tried using gc() to see how much memory is used at each step.
? Scanned Crawley R-Book and all other R books I have available and the FAQ
? on-line but no help really found.
? Running WinXP Pro (32 bit) with 4 GB RAM.
? One SATA drive pair is in RAID 0 configuration with 10000 MB allocated as
? virtual memory.
? I do have another machine set up with Ubuntu but it only has 2 GB RAM and
? have not been able to get R installed on that system.
? I can run smaller sample data sets w/o problems and everything plots as
? needed.
? However I need to review large data sets.
? Using latest R version 2.9.0 (2009-04-17)
? My ?data is in CSV format with a header row and is a big data set with
? 1,200,240 rows!
? E.g. below:
? Dur,TBC,Fmax,Fmin,Fmean,Fc,S1,Sc,
? 9.81,0,28.78,24.54,26.49,25.81,48.84,14.78,
? 4.79,1838.47,37.21,29.41,31.76,29.52,241.77,62.83,
? 4.21,5.42,28.99,26.23,27.53,27.4,76.03,11.44,
? 10.69,193.48,30.53,25.4,27.69,25.4,-208.19,26.05,
? 15.5,248.18,30.77,24.32,26.57,24.92,-202.76,18.64,
? 14.85,217.47,31.25,24.62,26.93,25.56,-88.4,10.32,
? 11.86,158.01,33.61,25.24,27.66,25.32,83.32,17.62,
? 14.05,229.74,30.65,24.24,26.76,25.24,61.87,14.06,
? 8.71,264.02,31.01,25.72,27.56,25.72,253.18,19.2,
? 3.91,10.3,25.32,24.02,24.55,24.02,-71.67,16.83,
? 16.11,242.21,29.85,24.02,26.07,24.62,79.45,19.11,
? 16.81,246.48,28.57,23.05,25.46,23.81,-179.82,15.95,
? 16.93,255.09,28.78,23.19,25.75,24.1,-112.21,16.38,
? 5.12,107.16,32,29.41,30.46,29.41,134.45,20.88,
? 16.7,150.49,27.97,22.92,24.91,23.95,42.96,16.81
? .... etc
? I am getting the following warning/error message:
? Error: cannot allocate vector of size 228.9 Mb
? Complete listing from R console below:
? > library(batcalls)
? Loading required package: ggplot2
? Loading required package: proto
? Loading required package: grid
? Loading required package: reshape
? Loading required package: plyr
? Attaching package: 'ggplot2'
? ? ? ? ? The following object(s) are masked from package:grid :
? ? ? ? ? ?nullGrob
? > gc()
? ? ? ? ? ?used (Mb) gc trigger (Mb) max used (Mb)
? Ncells 186251 ?5.0 ? ? 407500 10.9 ? 350000 ?9.4
? Vcells ?98245 ?0.8 ? ? 786432 ?6.0 ? 358194 ?2.8
? > BR <- read.csv ("C:/R-Stats/Bat calls/Reduced bats.csv")
? > gc()
? ? ? ? ? ? used (Mb) gc trigger ?(Mb) max used ?(Mb)
? Ncells ?188034 ?5.1 ? ? 667722 ?17.9 ? 378266 ?10.2
? Vcells 9733249 74.3 ? 20547202 156.8 20535538 156.7
? > attach(BR)
? > library(ggplot2)
? > library(MASS)
? > library(batcalls)
? > BRC<-kde2d(Sc,Fc)
? Error: cannot allocate vector of size 228.9 Mb
? > gc()
? ? ? ? ? ? ?used ?(Mb) gc trigger ?(Mb) ?max used ?(Mb)
? Ncells ? 198547 ? 5.4 ? ? 667722 ?17.9 ? ?378266 ?10.2
? Vcells 19339695 147.6 ?106768803 814.6 124960863 953.4
? >
? Tnx for any insight,
? Bruce
______________________________________________ R-help at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.