Skip to content

[R-pkg-devel] Winbuilder 'experimental' toolchain 4-5x slower than all other architectures

6 messages · Gábor Csárdi, Brian G. Peterson, Uwe Ligges +1 more

Original
Brian G. Peterson
# 
We've noticed over the past week that Winbuilder/R-Hub's 'experimental'
Rtools4 toolchain images for Windows are approximately 4-5x slower than
all other containers to build and check several packages we maintain.

This affects everything, including example timings, where this
architecture is the only architecture that shows any example run-times
over 5 seconds. (including Windows under the older toolchain).

Does CRAN expect us to optimize our code for the clearly non-performant 
'experimental' Windows toolchain?

I'd like to send a new version of our package PerformanceAnalytics to
CRAN, but the Windows experimental toolchain is preventing a 100% clean
set of checks.


Regards,

Brian
Gábor Csárdi
# 
Can you please show some numbers?

Thanks,
Gabor
On Wed, Feb 5, 2020 at 2:40 PM Brian G. Peterson <brian at braverock.com> wrote:
Brian G. Peterson
# 
For every other architecture, Travis-CI and R-Hub take approximately
10-12 minutes to build and check PerformanceAnalytics.  On the Windows
Rtools4 experimental toolchain, the build/check on Winbuilder and R-Hub 
take over an hour, sometimes as much as an hour and a half.

Non-trivial example timings are similarly 4-5x slower on the Windows
experimental toolchain.

Brian
On Wed, 2020-02-05 at 14:59 +0000, G?bor Cs?rdi wrote:
Uwe Ligges
# 
Then this is specific to your package, see the overall timings for the 
whole CRAN check:

https://cran.r-project.org/web/checks/check_timings.html

which is 2379156 sec for old and 2516120 sec for the new toolchain, i.e. 
roughly a 6%. Given the machines runs with very different loads it could 
even be the other way round.

And for PerformanceAnalytics I get overall 418 sec for old and 446 sec 
(without vignettes each) for the new toolchain (again, load may have 
varied):
https://cran.r-project.org/web/checks/check_results_PerformanceAnalytics.html



Best,
Uwe Ligges
On 05.02.2020 16:32, Brian G. Peterson wrote:
Uwe Ligges
#
On 05.02.2020 16:42, Uwe Ligges wrote:
I meant the new one, not the one on CRAN which is also similarly quick 
(less than 10% difference).

Best,
Uwe Ligges
see the overall timings for the
Dirk Eddelbuettel
#
On 5 February 2020 at 08:39, Brian G. Peterson wrote:
| We've noticed over the past week that Winbuilder/R-Hub's 'experimental'

Are you refering to winbuilder, or to rhub?

| Rtools4 toolchain images for Windows are approximately 4-5x slower than
| all other containers to build and check several packages we maintain.

That made me curious and I just did a quick run at rhub comparing windows
builds of r-devel (ie "old" toolchain) and r-devel using the new tools4. 

AFAICT the difference is due to more prebuilt packages coming in for the
former, and more packages building from source for the latter.

I used a small package with few depends---but critically with the
knitr/rmarkdown stack which leads to stringi building from source building
libicu first. That alone will set up back a good couple of minutes.  FWIW I
ended up with 4 minutes for the old tools versus 24 minutes with rtools4.

But as a cross-check, I used an even smaller package with no depends. And
there the time difference was neglible and both took about 3 minutes.

So I would recommend to try calling rhub _on that platform_ with the added

   check_args="--no-manual --no-vignettes"

argument so suppress vignette and manual builds and tests. I think that may
help you with shorter setup times.
 
| This affects everything, including example timings, where this
| architecture is the only architecture that shows any example run-times
| over 5 seconds. (including Windows under the older toolchain).
| 
| Does CRAN expect us to optimize our code for the clearly non-performant 
| 'experimental' Windows toolchain?
| 
| I'd like to send a new version of our package PerformanceAnalytics to
| CRAN, but the Windows experimental toolchain is preventing a 100% clean
| set of checks.

I think CRAN is fairly reasonable and lenient about run-time of examples, and
the local check time. Could you possibly demonstrate an actual slowdown in
example timings? If there is one, I am sure Jeroen would like to know more.

Dirk