Hi, As a project for a computer performance analysis paper I am taking this semester, I am going to look at the performance of the memory manager in R, with the aim of determining how fast it is and which areas most need improvement. The idea is that I will compare various versions of R, starting with 0.64.2, and then at a few stages in the implementation of the new memory management scheme (of which the first step was the hash tables for environments). I am looking for scripts to use as benchmarks for memory related activities. As a starting point I will use the tests distributed with R, but any other contributions would be helpful. I don't think the tests I have are particularly aimed at performance, but they should give a reasonable indication of which functions are the performance hogs. Any other suggestions would be greatly appreciated. Thanks, Lyndon -.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.- r-devel mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html Send "info", "help", or "[un]subscribe" (in the "body", not the subject !) To: r-devel-request@stat.math.ethz.ch _._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._
Memory profiling/benchmarking
3 messages · Lyndon Drake, Brian Ripley, Paul Gilbert
On Tue, 27 Jul 1999, Lyndon Drake wrote:
Hi, As a project for a computer performance analysis paper I am taking this semester, I am going to look at the performance of the memory manager in R, with the aim of determining how fast it is and which areas most need improvement. The idea is that I will compare various versions of R, starting with 0.64.2, and then at a few stages in the implementation of the new memory management scheme (of which the first step was the hash tables for environments). I am looking for scripts to use as benchmarks for memory related activities. As a starting point I will use the tests distributed with R, but any other contributions would be helpful. I don't think the tests I have are particularly aimed at performance, but they should give a reasonable indication of which functions are the performance hogs.
Currently the base package is treated specially (although less specially than it was, now we have hashing). Also, many of the most time-intensive things in R are done in compiled code. So some candidates for good tests are things done in large packages almost entirely in R. I used the V&R3 ch06.R script in doing some timings, as repeated linear regression (either stepwise selection or bootstrapping) looks like the right sort of thing. And on that the latest (in 0.65) memory management gave a speed up of around 50%.
Brian D. Ripley, ripley@stats.ox.ac.uk Professor of Applied Statistics, http://www.stats.ox.ac.uk/~ripley/ University of Oxford, Tel: +44 1865 272861 (self) 1 South Parks Road, +44 1865 272860 (secr) Oxford OX1 3TG, UK Fax: +44 1865 272595 -.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.- r-devel mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html Send "info", "help", or "[un]subscribe" (in the "body", not the subject !) To: r-devel-request@stat.math.ethz.ch _._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._
Lyndon There are several test scripts with my DSE library available at <http://www.bank-banque-canada.ca/pgilbert>. (Please don't make the mistake of trying to install the version on CRAN - it installs with R 0.61 but there have been a lot of changes to R since then.) The library calls compiled fortran for the really loop intensive core calculations, but this is fairly representative of R. With R (up to 0.64.2) the shortest set of tests (test.dse.lite) takes 7-8 minutes on a Sparc Ultra 10. (Splus 3.3 takes 4-5 minutes.) There are several longer tests, including some curvature calculations which take many hours. (In some of the more complicated tests there are a few things which do not work in R, but most things do.) I use generic functions a lot, and also use a set of "kernel" functions to help protect my code from ongoing changes in R and S. This means that function dispatch gets heavily exercised. Attached below are results from the light and medium test for 0.64.2 and the development snapshot (0.65) from a month ago (June 11). These show the timing at the bottom of each test. As I recall I had to make a small fix to 0.64.2 in order to get things working - but I don't remember what it was. If you have a problem, send me the error message and it will jog my memory. I don't see a large difference in the times, but I don't know if hashing was already in the snapshot I have. The pending changes in the ts library will have some affect on what has to be attached to run the tests. Hopefully I will get a chance to test the library today or tomorrow. I would like to see the results of your testing, even if you decide not to use these test. Paul Gilbert _________ On a sparc Ultra 10 R-0.64.2/ +++++++++++++++++++++++++++++++++++++++++++++++++ running R test.dse.lite +++++++++++++++++++++++++++++++++++++++++++++++++ RLIBS set to /home/mfa5/gilp/dse/pub/Rdse/rlibs
library("dse")
Attaching Package "package:syskern": The following object(s) are masked from package:dse : is.S is.Splus is.Svanilla rnorm The following object(s) are masked from package:base : RNGkind rnorm runif Attaching Package "package:tframe": The following object(s) are masked from package:base : [.ts summary.default ts tsp<- window.default window.ts Attaching Package "package:dse": The following object(s) are masked from package:base : [.ts summary.default ts tsp<- window.default window.ts
library("dse2")
Attaching Package "package:dsex1": The following object(s) are masked from package:base : curve
library("mva")
library("bats")
Attaching Package "package:bats": The following object(s) are masked from package:base : convolve
source("test.lite.s")
dse.1999.07alphaR Warning: This system is the by-product of on-going research. It has not been completely tested. Random number generator tests ...ok All tframe tests ... OK All dse1 (kernel) tests ... OK All dse2 tests ... OK All dse3 tests ... OK All dse4 tests ...Warning: skipping test 4 (requires stepwise). OK dse2 graphics tests ...Warning: Specified end later than end of data, end not changed. Warning: Specified end later than end of data, end not changed. Warning: Specified end later than end of data, end not changed. completed dse3 graphics tests ...completed dse4 graphics tests ...completed All Brief User Guide example part 1 tests ... OK All dse juice tests ...juice graphics tests ... OK OK
=========================================
R test.lite.s timing:
start Tuesday July 27 09:27:11 EDT 1999
end Tuesday July 27 09:34:59 EDT 1999
SunOS mfa99551 5.6 Generic_105181-11 sun4u sparc SUNW,Ultra-5_10
=========================================
+++++++++++++++++++++++++++++++++++++++++++++++++
running R test.dse.medium
+++++++++++++++++++++++++++++++++++++++++++++++++
x11()
library("dse", warn.conflicts=FALSE)
Attaching Package "package:syskern": The following object(s) are masked from package:dse : is.S is.Splus is.Svanilla rnorm The following object(s) are masked from package:base : RNGkind rnorm runif Attaching Package "package:tframe": The following object(s) are masked from package:base : [.ts summary.default ts tsp<- window.default window.ts
library("dse2", warn.conflicts=FALSE)
Attaching Package "package:dsex1": The following object(s) are masked from package:base : curve
library("opt")
library("bats")
Attaching Package "package:bats": The following object(s) are masked from package:base : convolve
source("test.medium.s")
dse.1999.07alphaR Warning: This system is the by-product of on-going research. It has not been completely tested. Brief User Guide example part 2 tests ...Warning: Specified end later than end of data, end not changed. Warning: Specified end later than end of data, end not changed. Warning: Specified end later than end of data, end not changed. Warning: Specified end later than end of data, end not changed. Warning: Specified end later than end of data, end not changed. Warning: Specified end later than end of data, end not changed. Warning: Specified end later than end of data, end not changed. Warning: Specified end later than end of data, end not changed. Warning: Specified end later than end of data, end not changed. Warning: Specified end later than end of data, end not changed. OK
=========================================
R test.medium.s timing:
start Tuesday July 27 09:35:00 EDT 1999
end Tuesday July 27 09:49:00 EDT 1999
SunOS mfa99551 5.6 Generic_105181-11 sun4u sparc SUNW,Ultra-5_10
=========================================
########################################################################
Devel version R 0.65 June 11, 1999 snapshot
+++++++++++++++++++++++++++++++++++++++++++++++++
running R test.dse.lite
+++++++++++++++++++++++++++++++++++++++++++++++++
RLIBS set to /home/mfa5/gilp/dse/pub/Rdse/rlibs
library("dse")
Attaching Package "package:syskern": The following object(s) are masked from package:dse : is.S is.Splus is.Svanilla rnorm Attaching Package "package:syskern": The following object(s) are masked from package:base : rnorm Attaching Package "package:tframe": The following object(s) are masked from package:base : [.ts summary.default ts tsp<- window.default window.ts Attaching Package "package:dse": The following object(s) are masked from package:base : [.ts summary.default ts tsp<- window.default window.ts
library("dse2")
Attaching Package "package:dsex1": The following object(s) are masked from package:base : curve
library("mva")
library("bats")
Attaching Package "package:bats": The following object(s) are masked from package:base : convolve
source("test.lite.s")
dse.1999.07alphaR Warning: This system is the by-product of on-going research. It has not been completely tested. Random number generator tests ...ok All tframe tests ... OK All dse1 (kernel) tests ... OK All dse2 tests ... OK All dse3 tests ... OK All dse4 tests ... OK dse2 graphics tests ...completed dse3 graphics tests ...completed dse4 graphics tests ...completed All Brief User Guide example part 1 tests ... OK All dse juice tests ...juice graphics tests ... OK OK Warning message: skipping test 4 (requires stepwise). in: dse4.function.tests(verbose = F, graphics = F)
=========================================
R test.lite.s timing:
start Tuesday July 27 09:59:15 EDT 1999
end Tuesday July 27 10:06:57 EDT 1999
SunOS mfa99551 5.6 Generic_105181-11 sun4u sparc SUNW,Ultra-5_10
=========================================
+++++++++++++++++++++++++++++++++++++++++++++++++
running R test.dse.medium
+++++++++++++++++++++++++++++++++++++++++++++++++
x11()
library("dse", warn.conflicts=FALSE)
Attaching Package "package:syskern": The following object(s) are masked from package:dse : is.S is.Splus is.Svanilla rnorm Attaching Package "package:syskern": The following object(s) are masked from package:base : rnorm Attaching Package "package:tframe": The following object(s) are masked from package:base : [.ts summary.default ts tsp<- window.default window.ts
library("dse2", warn.conflicts=FALSE)
Attaching Package "package:dsex1": The following object(s) are masked from package:base : curve
library("opt")
library("bats")
Attaching Package "package:bats": The following object(s) are masked from package:base : convolve
source("test.medium.s")
dse.1999.07alphaR Warning: This system is the by-product of on-going research. It has not been completely tested. Brief User Guide example part 2 tests ... OK
========================================= R test.medium.s timing: start Tuesday July 27 10:06:57 EDT 1999 end Tuesday July 27 10:20:47 EDT 1999 SunOS mfa99551 5.6 Generic_105181-11 sun4u sparc SUNW,Ultra-5_10 ========================================= -.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.- r-devel mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html Send "info", "help", or "[un]subscribe" (in the "body", not the subject !) To: r-devel-request@stat.math.ethz.ch _._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._