[R-pkg-devel] how to skip tests on CRAN but NOT on travis-ci?

I don't know if this is the best way to do it, but I've been using the covr package (https://github.com/jimhester/covr) to run all of my tests on codecov.io even when I use testthat::skip_on_cran(). The only downside is that it increases the amount of time needed for the package to build.

HTH,
Zhian

On Jun 21, 2015, at 07:46 , Jennifer Bryan <jenny at stat.ubc.ca> wrote:

I hope this is an appropriate to place to ask this. My question involves add-on tools and services, but I think they are in common use and others might have same question.

I am using testthat for testing and Travis (https://travis-ci.org) for continuous integration.

I do not want the vast majority of my tests to run on CRAN. They take a long time, require an internet connection, etc. I believe the official testthat solution for this is to use testthat::skip_on_cran() at the beginning of the body of the test.

However, I DO want these tests to run on Travis. In fact, I'd like to be as crazy strict on Travis as possible. More strict than on CRAN. So running R CMD check on Travis "as CRAN" is no good for me.

I think this coincides with this unanswered question on stackoverflow:
http://stackoverflow.com/questions/27557150/check-as-cran-but-do-not-skip-any-tests?lq=1

Thanks for any suggestions,
Jenny

Jennifer Bryan
Associate Professor
Department of Statistics and
  the Michael Smith Laboratories
University of British Columbia
Vancouver, BC Canada

______________________________________________
R-package-devel at r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel

Hi Jenny,

And a warm welcome here.  I hope you enjoy this place and its ability to form
questions (and answers) beyond the 140 char limit posed elsewhere. :-)

| I hope this is an appropriate to place to ask this. My question involves add-on tools and services, but I think they are in common use and others might have same question.
| 
| I am using testthat for testing and Travis (https://travis-ci.org) for continuous integration.
| 
| I do not want the vast majority of my tests to run on CRAN. They take a long time, require an internet connection, etc. I believe the official testthat solution for this is to use testthat::skip_on_cran() at the beginning of the body of the test.
| 
| However, I DO want these tests to run on Travis. In fact, I'd like to be as crazy strict on Travis as possible. More strict than on CRAN. So running R CMD check on Travis "as CRAN" is no good for me.
| 
| I think this coincides with this unanswered question on stackoverflow:
| http://stackoverflow.com/questions/27557150/check-as-cran-but-do-not-skip-any-tests?lq=1


Environment variables are perfect for this.  Eg in RcppRedis I can in fact
assume Redis to be present on Travis (as Travis has a webby-ish focus where
Redis is common) but am fairly certain win-builder and other machines do not.

So in tests/runUnitTests.R I have (with two typos I fixed here)



## if we know a redis server is set up, we can signal this -- cf .travis.yml
if (Sys.getenv("RunRcppRedisTests") == "yes") runTests <- TRUE

## here is a shortcut: if the user is 'edd' we also run tests
## ought to be wrong on CRAN, win-builder, ...
if (Sys.getenv("USER") == "edd") runTests <- TRUE



which shortly later is used in


## Tests for test run
if (runTests) {
    ## Run tests
    tests <- runTestSuite(testSuite)

    ....




This is generic, and you can surely adapt it from testthat.



In Rcpp we do something pretty similar within the set of test files in order
to remain within the one-minute limit.

So in tests/doRUnit.R we do set an environment variable to turn tests on,
else they are off by default:


## force tests to be executed if in dev release which we define as
## having a sub-release, eg 0.9.15.5 is one whereas 0.9.16 is not
if (length(strsplit(packageDescription("Rcpp")$Version, "\\.")[[1]]) > 3) {	
    if (Sys.getenv("RunAllRcppTests") != "no") { 				
        message("Setting \"RunAllRcppTests\"=\"yes\" for development release\n")
        Sys.setenv("RunAllRcppTests"="yes")
    }
}

    
Hope this helps,  Dirk

http://dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org

I think you can just use skip_on_cran() for the tests that you want to
skip on CRAN, and set the NOT_CRAN environment variable in your Travis
config.

Gabor

I hope this is an appropriate to place to ask this. My question involves add-on tools and services, but I think they are in common use and others might have same question.

I am using testthat for testing and Travis (https://travis-ci.org) for continuous integration.

I do not want the vast majority of my tests to run on CRAN. They take a long time, require an internet connection, etc. I believe the official testthat solution for this is to use testthat::skip_on_cran() at the beginning of the body of the test.

However, I DO want these tests to run on Travis. In fact, I'd like to be as crazy strict on Travis as possible. More strict than on CRAN. So running R CMD check on Travis "as CRAN" is no good for me.

I think this coincides with this unanswered question on stackoverflow:
http://stackoverflow.com/questions/27557150/check-as-cran-but-do-not-skip-any-tests?lq=1

Thanks for any suggestions,
Jenny

Jennifer Bryan
Associate Professor
Department of Statistics and
   the Michael Smith Laboratories
University of British Columbia
Vancouver, BC Canada

______________________________________________
R-package-devel at r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel

| Environment variables are perfect for this.  Eg in RcppRedis I can in fact
| assume Redis to be present on Travis (as Travis has a webby-ish focus where
| Redis is common) but am fairly certain win-builder and other machines do not.
| 
| So in tests/runUnitTests.R I have (with two typos I fixed here)
| 
| 
| 
| ## if we know a redis server is set up, we can signal this -- cf .travis.yml
| if (Sys.getenv("RunRcppRedisTests") == "yes") runTests <- TRUE
| 
| ## here is a shortcut: if the user is 'edd' we also run tests
| ## ought to be wrong on CRAN, win-builder, ...
| if (Sys.getenv("USER") == "edd") runTests <- TRUE
| 

And I forgot to show that in .travis.yml we do of course set this, and also
request a Redis server:


env:
  global:
    - _R_CHECK_FORCE_SUGGESTS_=FALSE
    - RunRcppRedisTests=yes

services:
  - redis-server



Dirk

 
| which shortly later is used in
| 
| 
| ## Tests for test run
| if (runTests) {
|     ## Run tests
|     tests <- runTestSuite(testSuite)
| 
|     ....
| 
| 
| 
| 
| This is generic, and you can surely adapt it from testthat.
| 
| 
| 
| In Rcpp we do something pretty similar within the set of test files in order
| to remain within the one-minute limit.
| 
| So in tests/doRUnit.R we do set an environment variable to turn tests on,
| else they are off by default:
| 
| 
| ## force tests to be executed if in dev release which we define as
| ## having a sub-release, eg 0.9.15.5 is one whereas 0.9.16 is not
| if (length(strsplit(packageDescription("Rcpp")$Version, "\\.")[[1]]) > 3) {	
|     if (Sys.getenv("RunAllRcppTests") != "no") { 				
|         message("Setting \"RunAllRcppTests\"=\"yes\" for development release\n")
|         Sys.setenv("RunAllRcppTests"="yes")
|     }
| }
| 
|     
| Hope this helps,  Dirk
| 
| 
| -- 
| http://dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org
| 
| ______________________________________________
| R-package-devel at r-project.org mailing list
| https://stat.ethz.ch/mailman/listinfo/r-package-devel

http://dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org

I hope this is an appropriate to place to ask this. My question involves add-on tools and services, but I think they are in common use and others might have same question.

I am using testthat for testing and Travis (https://travis-ci.org) for continuous integration.

I do not want the vast majority of my tests to run on CRAN. They take a long time, require an internet connection, etc. I believe the official testthat solution for this is to use testthat::skip_on_cran() at the beginning of the body of the test.

However, I DO want these tests to run on Travis. In fact, I'd like to be as crazy strict on Travis as possible. More strict than on CRAN. So running R CMD check on Travis "as CRAN" is no good for me.

I think this coincides with this unanswered question on stackoverflow:
http://stackoverflow.com/questions/27557150/check-as-cran-but-do-not-skip-any-tests?lq=1

Thanks for any suggestions,

Hi Jenny.

Besides the other suggestions, you can put tests in directories other
than "tests".  By default they won't run, but

R CMD check --test-dir=travisTests

will run them.  So if you can tell travis to add that option to the
command line, you could put all the travis-specific tests into
inst/travisTests, and it will run them instead of the regular ones.

Duncan Murdoch

On Sun, Jun 21, 2015 at 12:09 PM, Duncan Murdoch

Besides the other suggestions, you can put tests in directories other
than "tests".  By default they won't run, but

R CMD check --test-dir=travisTests

will run them.  So if you can tell travis to add that option to the
command line, you could put all the travis-specific tests into
inst/travisTests, and it will run them instead of the regular ones.

AFAIK this does not play very well with testthat, which the OP is clearly using.

Also, the OP wants run _additional_ tests, not something _instead_ of
the "regular" ones.

G.

[...]

Thanks for all of the responses!

What Gabor says below is true and is the path of least resistance in my particular case. As Dirk suggested, environment variables are the way to go.

If I put this into .travis.yml:

env:
  - NOT_CRAN=true

I get what I want

  * The Travis build still runs with the option --as-cran.
  * The tests harbouring testthat::skip_on_cran() are run locally and on Travis
  * The tests harbouring testthat::skip_on_cran() are NOT run on CRAN (at least, the little experiment I just ran on winbuilder suggests this ? haven't tried with actual CRAN submission)

TIL: the NOT_CRAN environment variable and the --as-cran option for R CMD check are completely independent of each other.

I also confirmed what Zhian said: if you're using skip_on_cran() and *don't* set NOT_CRAN=true in .travis.yml, you can still get some info about the affected tests in the coveralls result. But this doesn't seem a great solution, since you don't get much detail in the case of failure.

-- Jenny

I think you can just use skip_on_cran() for the tests that you want to
skip on CRAN, and set the NOT_CRAN environment variable in your Travis
config.

Gabor

On Sun, Jun 21, 2015 at 10:46 AM, Jennifer Bryan <jenny at stat.ubc.ca> wrote:

I hope this is an appropriate to place to ask this. My question involves add-on tools and services, but I think they are in common use and others might have same question.

I am using testthat for testing and Travis (https://travis-ci.org) for continuous integration.

I do not want the vast majority of my tests to run on CRAN. They take a long time, require an internet connection, etc. I believe the official testthat solution for this is to use testthat::skip_on_cran() at the beginning of the body of the test.

However, I DO want these tests to run on Travis. In fact, I'd like to be as crazy strict on Travis as possible. More strict than on CRAN. So running R CMD check on Travis "as CRAN" is no good for me.

I think this coincides with this unanswered question on stackoverflow:
http://stackoverflow.com/questions/27557150/check-as-cran-but-do-not-skip-any-tests?lq=1

Thanks for any suggestions,
Jenny

Jennifer Bryan
Associate Professor
Department of Statistics and
  the Michael Smith Laboratories
University of British Columbia
Vancouver, BC Canada

______________________________________________
R-package-devel at r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel

Jennifer Bryan
Associate Professor
Department of Statistics and
   the Michael Smith Laboratories
University of British Columbia
Vancouver, BC Canada

On Sun, Jun 21, 2015 at 12:09 PM, Duncan Murdoch
<murdoch.duncan at gmail.com> wrote:

Besides the other suggestions, you can put tests in directories other
than "tests".  By default they won't run, but

R CMD check --test-dir=travisTests

will run them.  So if you can tell travis to add that option to the
command line, you could put all the travis-specific tests into
inst/travisTests, and it will run them instead of the regular ones.

AFAIK this does not play very well with testthat, which the OP is clearly using.

Which seems like a problem with that package...

Also, the OP wants run _additional_ tests, not something _instead_ of
the "regular" ones.

But it's trivial to source files from the tests directory.  Or perhaps
(I don't know, I don't use it) it's trivial to tell travis to run the
standard R CMD check, followed by the suggested one.

Duncan Murdoch

On Sun, Jun 21, 2015 at 8:52 PM, Duncan Murdoch

[...]

But it's trivial to source files from the tests directory.

It is actually often not trivial at all. E.g. if you have a test with
setup/teardown, and the setup must be run before each test case, the
teardown after, and you want to run each test case in a clean
environment, this really does not play well with source-ing files.

It is definitely possible, and probably works for smaller packages
with simple tests, but imo it is not good practice.

Or perhaps
(I don't know, I don't use it) it's trivial to tell travis to run the
standard R CMD check, followed by the suggested one.

Yes, it is trivial, but I think this is not good practice either. E.g.
testthat gives you a summary in the end, and then you would need to
look at multiple summaries. It is easy to forget about the multiple
summaries, and miss one.

I think it is way easier if you just invoke the testing once, and let
it select what to run based on where it is running, what services are
available, etc.

G.

On Sun, Jun 21, 2015 at 8:52 PM, Duncan Murdoch
<murdoch.duncan at gmail.com> wrote:
[...]

But it's trivial to source files from the tests directory.

It is actually often not trivial at all. E.g. if you have a test with
setup/teardown, and the setup must be run before each test case, the
teardown after, and you want to run each test case in a clean
environment, this really does not play well with source-ing files.

It is definitely possible, and probably works for smaller packages
with simple tests, but imo it is not good practice.

Or perhaps
(I don't know, I don't use it) it's trivial to tell travis to run the
standard R CMD check, followed by the suggested one.

Yes, it is trivial, but I think this is not good practice either. E.g.
testthat gives you a summary in the end, and then you would need to
look at multiple summaries. It is easy to forget about the multiple
summaries, and miss one.

I think it is way easier if you just invoke the testing once, and let
it select what to run based on where it is running, what services are
available, etc.

As long as the reports tell you what tests were skipped, this seems
reasonable too.

Duncan Murdoch

Thanks for all of the responses!

What Gabor says below is true and is the path of least resistance in my particular case. As Dirk suggested, environment variables are the way to go.

If I put this into .travis.yml:

env:
  - NOT_CRAN=true

I get what I want

  * The Travis build still runs with the option --as-cran.
  * The tests harbouring testthat::skip_on_cran() are run locally and on Travis
  * The tests harbouring testthat::skip_on_cran() are NOT run on CRAN (at least, the little experiment I just ran on winbuilder suggests this ? haven't tried with actual CRAN submission)

TIL: the NOT_CRAN environment variable and the --as-cran option for R CMD check are completely independent of each other.

Yeah, NOT_CRAN is just a devtools convention - devtools sets not on
CRAN for you, and skip_on_cran() just tests for that env var (since I
have no control over what env vars are set on cran machines).

I think this used to work on travis because the old travis setup
called devtools::check(), while the new setup calls R CMD check
directly.

Hadley

http://had.co.nz/