[R-pkg-devel] checking windows timing?

FWIW, we dealt with the tests-taking-too-long problem for CVXR (which 
has a lot of tests) by mercilessly cutting out all tests on CRAN 
(testthat::skip_on_cran()) except for some fundamental ones on atoms, 
curvature, etc.? We then rely on Github actions (with NOT_CRAN = true) 
to run the full test suite on several platforms. Not perfect, but that's 
what worked for us.

-Naras

? Following up on my questions from yesterday:

? I've been advised to (drastically) reduce the checking time of my 
package on CRAN's Windows platform (currently at 23 min). I've gone 
through and added a lot of conditionals to skip tests on CRAN.? I 
could reduce it farther but I'm starting to encounter seriously 
diminishing returns, e.g.

?* do I condition out some particular test that takes 6 seconds? 
(e.g., a little slow, but not egregious)
?* do I put \dontrun{} around an example that takes 6 seconds [I'm a 
bit confused about this, but as I understand it CRAN checks with 
--run-donttest ? \dontrun{} also sometimes provokes complaints from 
CRAN, because "you should only use \dontrun{} for code that cannot be 
run by users" ...]

? I don't want to spend forever tweaking things, and I don't want to 
comment/condition out all of my tests, but I would really prefer to 
avoid bouncing from CRAN again. I don't know a *reliable* way to test 
whether CRAN-checking on Windows will take <10 mins or not ...

? I don't have a local Windows test platform.

? win-builder gives timings for long-running components, but not for 
the whole run.

? r-hub gives a time for the entire build (19 min in my case) - but 
this includes the time to set up the virtual machine, install all 
packages, etc..

? I can approximately predict that most examples and tests will take 
twice as long on CRAN's windows machines (and four times as long since 
tests, at least, are checked on both i386 and x86_64 architectures).

? Testing is currently taking 6 minutes on my local machine (newish 
Ubuntu 18.04 laptop), so I guess have more work to do, but I wonder if 
anyone has suggestions ...

? cheers
??? Ben Bolker

______________________________________________
R-package-devel at r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel

Ben Bolker 
    on Tue, 13 Oct 2020 12:46:11 -0400 writes:

>    Following up on my questions from yesterday:

    >    I've been advised to (drastically) reduce the checking
    > time of my package on CRAN's Windows platform (currently
    > at 23 min). I've gone through and added a lot of
    > conditionals to skip tests on CRAN.  I could reduce it
    > farther but I'm starting to encounter seriously
    > diminishing returns, e.g.


    >   * do I condition out some particular test that takes 6
    > seconds? (e.g., a little slow, but not egregious) * do I
    > put \dontrun{} around an example that takes 6 seconds [I'm
    > a bit confused about this, but as I understand it CRAN
    > checks with --run-donttest ? \dontrun{} also sometimes
    > provokes complaints from CRAN, because "you should only
    > use \dontrun{} for code that cannot be run by users" ...]

Really don't use \dontrun{}.  You'll stop running those examples
even for the users of  your package who  do

     example(<some>)

Rather do use an environment variable - based setup (as Uwe
mentioned).

In several packages of mine, I use

if(<pkg>:::doExtras()) { 
  ...  ## longer running examples / tests
}

sometimes also using withAutoprint({ .... })  which is important
when the longer running code is producing output that I'd want
to continue showing.


In my 'robustbase' package, I have somewhere in robustbase/R/<foo>.R

## Not exported, and only used because CRAN checks must be faster
doExtras <- function() {
    interactive() || nzchar(Sys.getenv("R_robustbase_check_extra")) ||
        identical("true", unname(Sys.getenv("R_PKG_CHECKING_doExtras")))
}

so doExtras()  is true for me,  *either* if I set the

package-specific        R_robustbase_check_extra  variable to non-empty
*or* the more global    R_PKG_CHECKING_doExtras   variable to non-empty.

That way, I can have all these check running in my own checks
(say by default, but can turn them off by unsetting the env.var.)

and yes, I use  doExtras  extensively in the tests, but very
much on purpose also in examples, there sometimes  combined with

 if(interactive()) { ... }

which keeps help page examples running interactively, but not in
'R CMD check' ..

Martin

    >    I don't want to spend forever tweaking things, and I
    > don't want to comment/condition out all of my tests, but I
    > would really prefer to avoid bouncing from CRAN again. I
    > don't know a *reliable* way to test whether CRAN-checking
    > on Windows will take <10 mins or not ...

    >    I don't have a local Windows test platform.

    >    win-builder gives timings for long-running components,
    > but not for the whole run.

    >    r-hub gives a time for the entire build (19 min in my
    > case) - but this includes the time to set up the virtual
    > machine, install all packages, etc..

    >    I can approximately predict that most examples and
    > tests will take twice as long on CRAN's windows machines
    > (and four times as long since tests, at least, are checked
    > on both i386 and x86_64 architectures).

    >    Testing is currently taking 6 minutes on my local
    > machine (newish Ubuntu 18.04 laptop), so I guess have more
    > work to do, but I wonder if anyone has suggestions ...

    >    cheers Ben Bolker