Skip to content

Benchmark

2 messages · Yoon, Hoon (CICG - NY Program Trading), Brian Ripley

Yoon, Hoon (CICG - NY Program Trading)
# 
Hello:

  I modified benchmark used in
http://www.informatik.uni-frankfurt.de/~stst/benchspl.txt now the attached
code will work with R, except for last test for some reason (Sorry ran out
of time to play with this).
  Immediate problem is that R does not show fraction of seconds. So, that
kinda makes comparison some what hard. I also looked at memory and CPU usage
on NT console. R seems to go into hybernation when not focused. The results
are mixed. I am sure if I add something complicated or/and loops that Splus
will slow down to crawl.
  Another interesting thing is that the random generator on Splus is a
rocket.
  I had many things running on the computer while trying to do this, so I am
very confident about results.
  Has anyone done something similar? If so let me know.

**************************************************************
S. Hoon Yoon                   (Quant)                    Merrill Lynch
Equity Trading 
yelled at yahoo.com hoon at bigfoot.com(w)
"Miracle is always only few standard deviations away, but so is
catastrophe."
* Expressed opinions are often my own, but NOT my employer's.
"I feel like a fugitive from the law of averages."    Maudin
**************************************************************

This was for Splus 4.5 on Pentium Pro 200, 256MB Compaq 2000 Ultra SCSI
I had a lot of stuff running concurrently, but shows some indication.
[1] "!!! S-Plus and R - Benchmarkprogram !!!"
[1] "================================"
[1] ""
[1] "If you use R, then do |R> dos.time <- system.time"
[1] " R does not currently show fraction of sec."
[1] " Modified Stefan's work by S. Hoon Yoon on Dec 1, 1998"
[1] " Visit his web site at
http://www.informatik.uni-frankfurt.de/~stst/benchspl.txt"
[1] " He has same bench for Gauss, Mathlab, Splus, Ox, etc... "
[1] ""
[1] ""
[1] "400 x 400 random matrix^1000 (sec.) : "
[1] 0.1899414
[1] "Eigenval. of a  300 x 300 randommatrix (sec.) :"
[1] 58.55005
[1] "Inverse of a  500 x 500  random matrix (sec.) :"
[1] 50.16992
[1] "500000  values sorted ascending (sec.) :"
[1] 2.800049
[1] "800 x 800 Toeplitzmatrix (sec.) :"
[1] 3.430176
[1] "Cholesky decomposition of a 500 x 500 -matrix (sec.) :"
[1] 5.599854
[1] "500 x 500 cross-product matrix (sec.) :"
[1] 20.11987
[1] "FFT over 100000 values (sec.) :"
[1] 1.379883
[1] "Gaussian error function over a 100000 x 100000 matrix (sec.) :"
[1] "Gamma function over a  600 x 600  matrix (sec.) :"
[1] 2.290039
[1] "Linear regression over a 500 x 500 matrix (sec.) :"
[1] 18.41016

R same machine
[1] "!!! S-Plus and R - Benchmarkprogram !!!"
[1] "================================"
[1] ""
[1] "If you use R, then do |R> dos.time <- system.time"
[1] " R does not currently show fraction of sec."
[1] " Sure would appreciate someone contributing that code"
[1] " Also interesting will be loop test and mem usage meter"
[1] " Modified Stefan's work by S. Hoon Yoon on Dec 1, 1998"
[1] " Visit his web site at
http://www.informatik.uni-frankfurt.de/~stst/benchspl.txt"
[1] " He has same bench for Gauss, Mathlab, Splus, Ox, etc... "
[1] ""
[1] ""
[1] "400 x 400 random matrix^1000 (sec.) : "
[1] NA NA  5 NA NA
[1] "Eigenval. of a  300 x 300 random matrix (sec.) :"
[1] NA NA 31 NA NA
[1] "Inverse of a  500 x 500  random matrix (sec.) :"
[1] NA NA 99 NA NA
[1] "5e+005  values sorted ascending (sec.) :"
[1] NA NA 21 NA NA
[1] "800 x 800 Toeplitzmatrix (sec.) :"
[1] NA NA  4 NA NA
[1] "Cholesky decomposition of a 500 x 500 -matrix (sec.) :"
[1] NA NA  2 NA NA
[1] "500 x 500 cross-product matrix (sec.) :"
[1] NA NA 16 NA NA
[1] "FFT over 1e+005 values (sec.) :"
[1] NA NA  1 NA NA
[1] "Gaussian error function over a 1e+005 x 1e+005 matrix (sec.) :"
[1] "Gamma function over a  600 x 600  matrix (sec.) :"
[1] NA NA  1 NA NA
[1] "Linear regression over a 500 x 500 matrix (sec.) :"
Error: 500 responses, but only 501 variables
Timing stopped at: NA NA 0 NA NA

  <<benchr.txt>> 
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: benchr.txt
Url: https://stat.ethz.ch/pipermail/r-help/attachments/19981201/c66c42c1/benchr.txt
Brian Ripley
#
On Tue, 1 Dec 1998, Yoon, Hoon (CICG - NY Program Trading) wrote:

            

      
      
      
    

        
You really do need to say that you did this on Wintel, and give versions of
the packages you used.  As R does not have a BATCH mode that runs on NT
AFAIK, I assume you source-ed the code, which is not a good idea and can be
most misleading due to different memory usage from typing at the command
line.  In short, with benchmarks the details do matter greatly.

            

      
      
      
    

        
No, it isn't!  As RNGs go it is quite slow, but the R one is very slow
(despite the title of the paper `efficient' for which they  never gave any 
evidence).  Lots of timings in a paper of mine written a decade ago:

Ripley, B. D. (1990) Thoughts on pseudorandom number generators. J. Comput.
Appl. Math. 31, 153-163. 

Not that in many real tasks you will notice the difference: there are some
but I would not do those in R.

            

      
      
      
    

        
These sort of tests do not test the language nor typical uses of it.
They test a few of the built-in routines, thereby mainly testing the
compilers used but also the accuracy/speed compromises used. Thus for
example for an eigendecomposition I do want to know not only the speed but
also the accuracy.

I have run fairly extensive tests of real code (from V&R, for example),
mainly to find problems.  I won't give you extensive sets of numbers, but
re-iterate what I said recently on R-devel: the times are almost always
within a factor of two, and on Solaris S-PLUS 3.4 is usually a little
faster than R 0.63.0. (I am not going to comment on something as
experimental as R on Windows as although I do now have a batch mode running
reliably on Win9x, this is very recent.)

Note that, having tried it, I do NOT believe `if I add something
complicated or/and loops that Splus will slow down to crawl'. Your
prejudices are showing!