Hello everyone, I'm sorry to bother you, but I would like some help getting a package ( *suppdata*; https://github.com/ropensci/suppdata) available through CRAN. The package downloads supplementary materials from papers and data repositories on the basis of their DOI, making reproducible analyses a little easier to share and create. The package's unit tests involve downloading data, and when multiple requests for the same resource are sent from a single machine (as seems to be done by CRAN's testing servers) the data hosts will sometimes refuse the connection and so the package's tests will fail. I emphasise that the tests pass at Travis-CI (https://travis-ci.org/ropensci/suppdata) and OpenCPU ( https://ropensci.ocpu.io/suppdata/info); I am confident this is a connection issue but I would be grateful to be shown I am wrong. I am not sure how to solve this problem, and was hoping you might have some advice. Some things I have considered include: 1. Skipping all unit tests on CRAN (using something like *testtht::skip_on_cran*). This would immediately fix the problem, and as a mitigating factor we report automated test result and coverage on the package's GitHub page (https://github.com/ropensci/suppdata). 2. Using HTTP-mocking to avoid requiring a call to a server during tests at all. I would be uncomfortable relying solely on this for all tests, since if the data hosters changed things we wouldn't know. Thus I would still want the Internet-enabled tests, which would also have to be turned off for CRAN (see 1 above). This would also be a lot of additional work. 3. Somehow bypassing the requirement for the unit tests to all pass before the package is checked by the CRAN maintainers. I have no idea if this is something CRAN would be willing to do, or if it is even possible. It would be the easiest option for me, but I don't want to create extra work for other people! 4. Slowing the tests with something like *Sys.sleep*. This might work, but would slow the tests massively and so might that cause problems for CRAN? Does anyone have any advice as to which of the above would be the best option, or who I should email directly about this? Thank you for your help, and I apologise if I've missed some aspect of the documentation that already explains how to solve this, Will Pearse --- Need a phylogeny? Try phyloGenerator: original <http://willpearse.github.io/phyloGenerator/> or new version <http://willpearse.github.io/phyloGenerator2/> Measuring phylogenetic structure? Try install.packages('pez') Will Pearse <http://www.pearselab.com/> Assistant Professor of Biology, Utah State University Office: +1-435-797-0831; Room LSB-333 Skype: will.pearse
[R-pkg-devel] Submitting a package whose unit tests sometimes fail because of server connections
6 messages · Will, Steven Scott, Dirk Eddelbuettel +2 more
I'd mock the tests you want run on Cran and keep live fire tests that you can run manually. Just don't include the live fire stuff in the package.
On Tue, Apr 16, 2019, 10:57 AM Will <will.pearse at gmail.com> wrote:
Hello everyone, I'm sorry to bother you, but I would like some help getting a package ( *suppdata*; https://github.com/ropensci/suppdata) available through CRAN. The package downloads supplementary materials from papers and data repositories on the basis of their DOI, making reproducible analyses a little easier to share and create. The package's unit tests involve downloading data, and when multiple requests for the same resource are sent from a single machine (as seems to be done by CRAN's testing servers) the data hosts will sometimes refuse the connection and so the package's tests will fail. I emphasise that the tests pass at Travis-CI (https://travis-ci.org/ropensci/suppdata) and OpenCPU ( https://ropensci.ocpu.io/suppdata/info); I am confident this is a connection issue but I would be grateful to be shown I am wrong. I am not sure how to solve this problem, and was hoping you might have some advice. Some things I have considered include: 1. Skipping all unit tests on CRAN (using something like *testtht::skip_on_cran*). This would immediately fix the problem, and as a mitigating factor we report automated test result and coverage on the package's GitHub page (https://github.com/ropensci/suppdata). 2. Using HTTP-mocking to avoid requiring a call to a server during tests at all. I would be uncomfortable relying solely on this for all tests, since if the data hosters changed things we wouldn't know. Thus I would still want the Internet-enabled tests, which would also have to be turned off for CRAN (see 1 above). This would also be a lot of additional work. 3. Somehow bypassing the requirement for the unit tests to all pass before the package is checked by the CRAN maintainers. I have no idea if this is something CRAN would be willing to do, or if it is even possible. It would be the easiest option for me, but I don't want to create extra work for other people! 4. Slowing the tests with something like *Sys.sleep*. This might work, but would slow the tests massively and so might that cause problems for CRAN? Does anyone have any advice as to which of the above would be the best option, or who I should email directly about this? Thank you for your help, and I apologise if I've missed some aspect of the documentation that already explains how to solve this, Will Pearse --- Need a phylogeny? Try phyloGenerator: original <http://willpearse.github.io/phyloGenerator/> or new version <http://willpearse.github.io/phyloGenerator2/> Measuring phylogenetic structure? Try install.packages('pez') Will Pearse <http://www.pearselab.com/> Assistant Professor of Biology, Utah State University Office: +1-435-797-0831; Room LSB-333 Skype: will.pearse [[alternative HTML version deleted]]
______________________________________________ R-package-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
On 16 April 2019 at 11:40, Will wrote:
| Some things I have considered include: | | 1. Skipping all unit tests on CRAN (using something like | *testtht::skip_on_cran*). This would immediately fix the problem, and as | a mitigating factor we report automated test result and coverage on the | package's GitHub page (https://github.com/ropensci/suppdata). | 2. Using HTTP-mocking to avoid requiring a call to a server during tests | at all. I would be uncomfortable relying solely on this for all tests, | since if the data hosters changed things we wouldn't know. Thus I would | still want the Internet-enabled tests, which would also have to be turned | off for CRAN (see 1 above). This would also be a lot of additional work. | 3. Somehow bypassing the requirement for the unit tests to all pass | before the package is checked by the CRAN maintainers. I have no idea if | this is something CRAN would be willing to do, or if it is even possible. | It would be the easiest option for me, but I don't want to create extra | work for other people! | 4. Slowing the tests with something like *Sys.sleep*. This might work, | but would slow the tests massively and so might that cause problems for | CRAN? | | Does anyone have any advice as to which of the above would be the best | option, or who I should email directly about this? 5. Run a hybrid scheme where you have multiple levels: 5.1 Do what eg Rcpp does and only opt into 'all tests' when an overall variable is set; that variable can be set conveniently in .travis.yml and conditionally in your test runner below ~/tests/ That way you can skip tests that would fail. 5.2 Do a lot of work and wrap 3. above into try() / tryCatch() and pass if _your own aggregation of tests_ passes a threshold. Overkill to me. 5.3 Turn all tests on / off based on some other toggle. I.e. I don't think I test all features of RcppRedis on CRAN as I can't assume a redis server, but I do run those tests at home, on Travis, ... Overall, I would recommend to 'keep it simple & stupid' (KISS) as life is to short to argue^Hdebate this with CRAN. And their time is too precious so we should try to make their life easier. Dirk
http://dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org
On Tue, 16 Apr 2019 at 19:57, Will <will.pearse at gmail.com> wrote:
Hello everyone, I'm sorry to bother you, but I would like some help getting a package ( *suppdata*; https://github.com/ropensci/suppdata) available through CRAN. The package downloads supplementary materials from papers and data repositories on the basis of their DOI, making reproducible analyses a little easier to share and create. The package's unit tests involve downloading data, and when multiple requests for the same resource are sent from a single machine (as seems to be done by CRAN's testing servers) the data hosts will sometimes refuse the connection and so the package's tests will fail. I emphasise that the tests pass at Travis-CI (https://travis-ci.org/ropensci/suppdata) and OpenCPU ( https://ropensci.ocpu.io/suppdata/info); I am confident this is a connection issue but I would be grateful to be shown I am wrong. I am not sure how to solve this problem, and was hoping you might have some advice. Some things I have considered include: 1. Skipping all unit tests on CRAN (using something like *testtht::skip_on_cran*). This would immediately fix the problem, and as a mitigating factor we report automated test result and coverage on the package's GitHub page (https://github.com/ropensci/suppdata).
This doesn't seem a good idea.
2. Using HTTP-mocking to avoid requiring a call to a server during tests at all. I would be uncomfortable relying solely on this for all tests, since if the data hosters changed things we wouldn't know. Thus I would still want the Internet-enabled tests, which would also have to be turned off for CRAN (see 1 above). This would also be a lot of additional work.
What about mocking on CRAN and run Internet-enabled tests otherwise?
3. Somehow bypassing the requirement for the unit tests to all pass before the package is checked by the CRAN maintainers. I have no idea if this is something CRAN would be willing to do, or if it is even possible. It would be the easiest option for me, but I don't want to create extra work for other people!
I think it's acceptable to skip a test if something is not available, but not the majority of them, for packages like this which basically is about downloading stuff from APIs. Mocking on CRAN, as said before, is the way to go here.
4. Slowing the tests with something like *Sys.sleep*. This might work, but would slow the tests massively and so might that cause problems for CRAN?
This is not a good idea either.
I?aki ?car
5.4 In the spirit of simple & stupid you can also use the built in mechanism for doing this: organize some of your tests in subdirectories like inst/testWithInternet, inst/veryLongTests, inst/testsNeedingLicence, inst/testsNeedingSpecialCluster, etc. CRAN will only run the tests in the tests/ directory, but you can check them yourself using R CMD check --test-dir=inst/testWithInternet whatever.tar.gz
> In a separate response On 4/16/19 2:06 PM, Steven Scott wrote:
> Just don't include the live fire stuff in the package. Please do not do this. If you omit tests from your package then it cannot be properly checked by other people. Paul Gilbert
On 4/16/19 2:16 PM, Dirk Eddelbuettel wrote:
On 16 April 2019 at 11:40, Will wrote: | Some things I have considered include: | | 1. Skipping all unit tests on CRAN (using something like | *testtht::skip_on_cran*). This would immediately fix the problem, and as | a mitigating factor we report automated test result and coverage on the | package's GitHub page (https://github.com/ropensci/suppdata). | 2. Using HTTP-mocking to avoid requiring a call to a server during tests | at all. I would be uncomfortable relying solely on this for all tests, | since if the data hosters changed things we wouldn't know. Thus I would | still want the Internet-enabled tests, which would also have to be turned | off for CRAN (see 1 above). This would also be a lot of additional work. | 3. Somehow bypassing the requirement for the unit tests to all pass | before the package is checked by the CRAN maintainers. I have no idea if | this is something CRAN would be willing to do, or if it is even possible. | It would be the easiest option for me, but I don't want to create extra | work for other people! | 4. Slowing the tests with something like *Sys.sleep*. This might work, | but would slow the tests massively and so might that cause problems for | CRAN? | | Does anyone have any advice as to which of the above would be the best | option, or who I should email directly about this? 5. Run a hybrid scheme where you have multiple levels: 5.1 Do what eg Rcpp does and only opt into 'all tests' when an overall variable is set; that variable can be set conveniently in .travis.yml and conditionally in your test runner below ~/tests/ That way you can skip tests that would fail. 5.2 Do a lot of work and wrap 3. above into try() / tryCatch() and pass if _your own aggregation of tests_ passes a threshold. Overkill to me. 5.3 Turn all tests on / off based on some other toggle. I.e. I don't think I test all features of RcppRedis on CRAN as I can't assume a redis server, but I do run those tests at home, on Travis, ... Overall, I would recommend to 'keep it simple & stupid' (KISS) as life is to short to argue^Hdebate this with CRAN. And their time is too precious so we should try to make their life easier. Dirk
2 days later
Hello everyone, Thank you very much for your help with this! These are some excellent ideas; I think we will go with either the mocking approach or a variant of Dirk's suggestion to use a test threshold. Thanks again! Will --- Need a phylogeny? Try phyloGenerator: original <http://willpearse.github.io/phyloGenerator/> or new version <http://willpearse.github.io/phyloGenerator2/> Measuring phylogenetic structure? Try install.packages('pez') Will Pearse <http://www.pearselab.com/> Assistant Professor of Biology, Utah State University Office: +1-435-797-0831; Room LSB-333 Skype: will.pearse
On Tue, 16 Apr 2019 at 12:20, I?aki Ucar <iucar at fedoraproject.org> wrote:
On Tue, 16 Apr 2019 at 19:57, Will <will.pearse at gmail.com> wrote:
Hello everyone, I'm sorry to bother you, but I would like some help getting a package ( *suppdata*; https://github.com/ropensci/suppdata) available through
CRAN.
The package downloads supplementary materials from papers and data repositories on the basis of their DOI, making reproducible analyses a little easier to share and create. The package's unit tests involve downloading data, and when multiple requests for the same resource are sent from a single machine (as seems
to
be done by CRAN's testing servers) the data hosts will sometimes refuse
the
connection and so the package's tests will fail. I emphasise that the
tests
pass at Travis-CI (https://travis-ci.org/ropensci/suppdata) and OpenCPU
(
https://ropensci.ocpu.io/suppdata/info); I am confident this is a connection issue but I would be grateful to be shown I am wrong. I am not sure how to solve this problem, and was hoping you might have some
advice.
Some things I have considered include: 1. Skipping all unit tests on CRAN (using something like *testtht::skip_on_cran*). This would immediately fix the problem, and
as
a mitigating factor we report automated test result and coverage on
the
package's GitHub page (https://github.com/ropensci/suppdata).
This doesn't seem a good idea.
2. Using HTTP-mocking to avoid requiring a call to a server during
tests
at all. I would be uncomfortable relying solely on this for all tests, since if the data hosters changed things we wouldn't know. Thus I
would
still want the Internet-enabled tests, which would also have to be
turned
off for CRAN (see 1 above). This would also be a lot of additional
work. What about mocking on CRAN and run Internet-enabled tests otherwise?
3. Somehow bypassing the requirement for the unit tests to all pass before the package is checked by the CRAN maintainers. I have no idea
if
this is something CRAN would be willing to do, or if it is even
possible.
It would be the easiest option for me, but I don't want to create
extra
work for other people!
I think it's acceptable to skip a test if something is not available, but not the majority of them, for packages like this which basically is about downloading stuff from APIs. Mocking on CRAN, as said before, is the way to go here.
4. Slowing the tests with something like *Sys.sleep*. This might work, but would slow the tests massively and so might that cause problems
for
CRAN?
This is not a good idea either. -- I?aki ?car