R run time question

-----Original Message-----
From: r-sig-mac-bounces at r-project.org [mailto:r-sig-mac-bounces at r-project.org] On Behalf Of Wayne Gray
Sent: July-13-11 12:25 PM
To: r-sig-mac at r-project.org
Subject: [R-SIG-Mac] R run time question

All,

We are doing ANOVAs that take a long time (> 3 hrs) to run in 64-bit mode on an Intel MacPro with 8
gbytes and 2.66 GHZ Intel Core i7 machine.

We have just tried running these on our server where we don't care as much how long it takes. The
server is a Mac "2 x 3 GHz Dual-Core Intel Xeon" with 13 GB 667 MHz DDR2 FB-DIMM.

We have installed R "R 2.10.1 GUI 1.31-np Tiger build 32-bit (5538)" on the server.

Why are you installing a 32 bit R?  And why an older version of R?

Can you not install a 64 bit version of the latest R?

The data.frame is composed of the following observations, factors, and numerals:

str(e1jit41blkAll)

'data.frame':	21648 obs. of  6 variables:
$ subjectid: Factor w/ 22 levels "1379","1744",..: 1 1 1 1 1 1 1 1 1 1 ...
$ cond     : Factor w/ 2 levels "Visual","Auditory": 1 1 1 1 1 1 1 1 1 1 ...
$ block    : Factor w/ 8 levels "2","3","4","5",..: 1 1 1 1 1 1 1 1 1 1 ...
$ cbtime   : Factor w/ 41 levels "1","2","3","4",..: 1 10 11 12 13 14 15 16 17 18 ...
$ dirtime  : Ord.factor w/ 3 levels "Early"<"Middle"<..: 1 1 1 1 1 1 1 1 1 1 ...
$ jitter   : num  7.78 7.33 5.56 5 6.11 ...

When we run the following ANOVA on this dataframe:

anova.ALL.e1jit41.blkAll  <- with(e1jit41blkAll, aov(jitter ~ cond*block*dirtime*cbtime +

error(subjectid/(block*dirtime*cbtime)), data = e1jit41blkAll))

We get the following feedback:

Error: cannot allocate vector of size 3.5 Gb

Running the ANOVA requires more than 4GB of RAM, which can not be done
with 32 bit versions of software.  So you will need 64 bit R to do this analysis.

R(29720,0xa000d000) malloc: *** vm_allocate(size=3749089280) failed (error code=3)
R(29720,0xa000d000) malloc: *** error: can't allocate region
R(29720,0xa000d000) malloc: *** set a breakpoint in szone_error to debug
R(29720,0xa000d000) malloc: *** vm_allocate(size=3749089280) failed (error code=3)
R(29720,0xa000d000) malloc: *** error: can't allocate region
R(29720,0xa000d000) malloc: *** set a breakpoint in szone_error to debug

So two categories of questions for all of you Mac-R wizards out there.

First category of question: can this analysis be run faster (and if so, how) on our laptop Intel Macs?

Second category of question: is there anything that can be done to the Apple Server or to our analysis
so that the analysis runs on our server?

What OS version is on your server?  If it's an old 32 bit version, upgrading the server
to a recent 64 bit OS will help.  Then install a recent 64 bit version of R.

If you can do this, you can better assess whether this analysis can be done 
faster on the 8GB laptops.

R runs entirely in RAM, so if the analysis needs more than 8GB of RAM, you may
be experiencing "swapping" on the laptops.  The operating system allocates
as much memory as the job needs - if that amount of memory exceeds the amount
of RAM you have, then the rest is allocated to "virtual memory" which is
on the disk drive.  Swapping occurs when part of the virtual memory of your job
that is on disk is needed - the OS has to copy a chunk of what is in RAM out to
disk, so it can copy the other chunk on disk back into RAM.

Monitor the laptop with a shell command line process such as "top" 
and you will be able to determine how much virtual memory your R job
is using, and how many "swaps" are involved.  Swapping really slows things
down as reading and writing from the disk is slow.  If your job is swapping
on the laptop, but uses less than 13GB of memory in total, it will run faster
on the server.  If the job is not swapping at all on the laptop,  it will not
run appreciably faster on the server.


HTH

Steve McKinney

Many thanks,

Wayne Gray

	[[alternative HTML version deleted]]