----- Original Message ---- From: xiaoming gu <xiaoming.gu at gmail.com> To: r-sig-hpc at r-project.org Sent: Tue, April 6, 2010 7:45:05 PM Subject: [R-sig-hpc] looking for available R benchmarks Hi, all. I'm a stranger in R and wondering whether there are some available standard benchmarks for R. I would like to know some workloads written in R and designed for the purpose similar to SPEC CPU2006. And each workload has several different input levels. Thanks very much. Xiaoming [[alternative HTML version deleted]] _______________________________________________ It might be helpful if you could clarify the intent of your question. Broadly speaking, there are two motivations for benching: a) Comparing the performance of different processors (the dependent variable) running the same R code (independent variable) b) Comparing the performance of different R codes (dependent variable) running on the same processor (independent variable) Approach (a) is the motivation behind the SPEC benching cartel. As far as I'm aware, the SPEC b/m doesn't contain any R codes but even if it did, their choice of code would not necessarily be the best choice for your purposes. You could, however, apply the same idea using R codes that you know and love. Although this is easy to say, it's not necessarily easy to do correctly. It's very easy to do a lot of work that is meaningless. That's one reason SPEC pre-packages the runtime codes and analysis. In approach (b), you compare code performance (e.g., execution times) on a fixed choice of processor, e.g., a machine that has already been purchased. That's what is reported here http://www.revolution-computing.com/products/benchmarks.php The test rig is fixed as a Dell XPS Studio PC with 2.40GHz Intel Core 2 Quad processors, 6GB system RAM running Windows Server 2008. They then compare the performance of their REvolution R codes with R-2.7.2 from CRAN. Although your question would appear to be closer to (a) in intent, it's not clear to me that (b) might not be the better choice. I would need to know more. Also, I'm not sure what you mean by "different input levels." --Neil Gunther ________________________________________________________________________ TAKING THE PITH OUT OF PERFORMANCE http://perfdynamics.blogspot.com/ Follow me on Twitter http://twitter.com/DrQz PERFORMANCE DYNAMICS COMPANY http://www.perfdynamics.com/ Castro Valley, CA 94552
looking for available R benchmarks
3 messages · Red Roo, xiaoming gu
An embedded and charset-unspecified text was scrubbed... Name: not available URL: <https://stat.ethz.ch/pipermail/r-sig-hpc/attachments/20100407/7ee873c2/attachment.pl>
1 day later
From: xiaoming gu <xiaoming.gu at gmail.com>
Cc: r-sig-hpc at r-project.org Sent: Wed, April 7, 2010 9:22:58 AM Subject: Re: [R-sig-hpc] looking for available R benchmarks ----- Original Message ---- >>From: xiaoming gu <xiaoming.gu at gmail.com> >>To: r-sig-hpc at r-project.org >>Sent: Tue, April 6, 2010 7:45:05 PM >>Subject: [R-sig-hpc] looking for available R benchmarks > >>Hi, all. I'm a stranger in R and wondering whether there are some available >>standard benchmarks for R. I would like to know some workloads written in R >>and designed for the purpose similar to SPEC CPU2006. And each workload has >>several different input levels. Thanks very much. > > >Xiaoming [[alternative HTML version deleted]] >>_______________________________________________ > >It might be helpful if you could clarify the intent of your question. > >>Broadly speaking, there are two motivations for benching: >>a) Comparing the performance of different processors (the dependent variable) running the same R code (independent variable) > >>b) Comparing the performance of different R codes (dependent variable) running on the same processor (independent variable) > >>Approach (a) is the motivation behind the SPEC benching cartel. As far as I'm aware, the SPEC b/m doesn't contain any R codes but even if it did, their choice of code would not necessarily be the best choice for your purposes. You could, however, apply the same idea using R codes that you know and love. Although this is easy to say, it's not necessarily easy to do correctly. It's very easy to do a lot of work that is meaningless. That's one reason SPEC pre-packages the runtime codes and analysis. > >>In approach (b), you compare code performance (e.g., execution times) on a fixed choice of processor, e.g., a machine that has already been purchased. That's what is reported here http://www.revolution-computing.com/products/benchmarks.php The test rig is fixed as a Dell XPS Studio PC with 2.40GHz Intel Core 2 Quad processors, 6GB system RAM running Windows Server 2008. They then compare the performance of their REvolution R codes with R-2.7.2 from CRAN. > >>Although your question would appear to be closer to (a) in intent, it's not clear to me that (b) might not be the better choice. I would need to know more. > >>Also, I'm not sure what you mean by "different input levels." > >>--Neil Gunther > Thanks for all your kindly replies. My intention is closer to (b). I'm trying http://r.research.att.com/benchmarks/ and REvolution R. "different input levels" is typically about running time. For example, a very large input makes program run several hours but a small input only takes less than one minute. With this feature, I can study R program performance in a more flexible way. Xiaoming >>________________________________________________________________________ > Case in point for benching approach (b): Comparative performance of different R implementations for table-lookups on a fixed test platform, viz., elapsed times using system.time() on a Macbook with 2 GHz, and 4 GB RAM. http://broadcast.oreilly.com/2010/03/lookup-performance-in-r.html --njg ________________________________________________________________________ TAKING THE PITH OUT OF PERFORMANCE http://perfdynamics.blogspot.com/ Follow me on Twitter http://twitter.com/DrQz PERFORMANCE DYNAMICS COMPANY http://www.perfdynamics.com/ Castro Valley, CA 94552