R run time question
-----Original Message----- From: r-sig-mac-bounces at r-project.org [mailto:r-sig-mac-bounces at r-project.org] On Behalf Of Wayne Gray Sent: July-13-11 12:25 PM To: r-sig-mac at r-project.org Subject: [R-SIG-Mac] R run time question All, We are doing ANOVAs that take a long time (> 3 hrs) to run in 64-bit mode on an Intel MacPro with 8 gbytes and 2.66 GHZ Intel Core i7 machine. We have just tried running these on our server where we don't care as much how long it takes. The server is a Mac "2 x 3 GHz Dual-Core Intel Xeon" with 13 GB 667 MHz DDR2 FB-DIMM. We have installed R "R 2.10.1 GUI 1.31-np Tiger build 32-bit (5538)" on the server.
Why are you installing a 32 bit R? And why an older version of R? Can you not install a 64 bit version of the latest R?
The data.frame is composed of the following observations, factors, and numerals:
str(e1jit41blkAll)
'data.frame': 21648 obs. of 6 variables: $ subjectid: Factor w/ 22 levels "1379","1744",..: 1 1 1 1 1 1 1 1 1 1 ... $ cond : Factor w/ 2 levels "Visual","Auditory": 1 1 1 1 1 1 1 1 1 1 ... $ block : Factor w/ 8 levels "2","3","4","5",..: 1 1 1 1 1 1 1 1 1 1 ... $ cbtime : Factor w/ 41 levels "1","2","3","4",..: 1 10 11 12 13 14 15 16 17 18 ... $ dirtime : Ord.factor w/ 3 levels "Early"<"Middle"<..: 1 1 1 1 1 1 1 1 1 1 ... $ jitter : num 7.78 7.33 5.56 5 6.11 ... When we run the following ANOVA on this dataframe:
anova.ALL.e1jit41.blkAll <- with(e1jit41blkAll, aov(jitter ~ cond*block*dirtime*cbtime +
error(subjectid/(block*dirtime*cbtime)), data = e1jit41blkAll)) We get the following feedback: Error: cannot allocate vector of size 3.5 Gb
Running the ANOVA requires more than 4GB of RAM, which can not be done with 32 bit versions of software. So you will need 64 bit R to do this analysis.
R(29720,0xa000d000) malloc: *** vm_allocate(size=3749089280) failed (error code=3) R(29720,0xa000d000) malloc: *** error: can't allocate region R(29720,0xa000d000) malloc: *** set a breakpoint in szone_error to debug R(29720,0xa000d000) malloc: *** vm_allocate(size=3749089280) failed (error code=3) R(29720,0xa000d000) malloc: *** error: can't allocate region R(29720,0xa000d000) malloc: *** set a breakpoint in szone_error to debug
So two categories of questions for all of you Mac-R wizards out there. First category of question: can this analysis be run faster (and if so, how) on our laptop Intel Macs? Second category of question: is there anything that can be done to the Apple Server or to our analysis so that the analysis runs on our server?
What OS version is on your server? If it's an old 32 bit version, upgrading the server to a recent 64 bit OS will help. Then install a recent 64 bit version of R. If you can do this, you can better assess whether this analysis can be done faster on the 8GB laptops. R runs entirely in RAM, so if the analysis needs more than 8GB of RAM, you may be experiencing "swapping" on the laptops. The operating system allocates as much memory as the job needs - if that amount of memory exceeds the amount of RAM you have, then the rest is allocated to "virtual memory" which is on the disk drive. Swapping occurs when part of the virtual memory of your job that is on disk is needed - the OS has to copy a chunk of what is in RAM out to disk, so it can copy the other chunk on disk back into RAM. Monitor the laptop with a shell command line process such as "top" and you will be able to determine how much virtual memory your R job is using, and how many "swaps" are involved. Swapping really slows things down as reading and writing from the disk is slow. If your job is swapping on the laptop, but uses less than 13GB of memory in total, it will run faster on the server. If the job is not swapping at all on the laptop, it will not run appreciably faster on the server. HTH Steve McKinney
Many thanks, Wayne Gray [[alternative HTML version deleted]]
_______________________________________________ R-SIG-Mac mailing list R-SIG-Mac at r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-mac