Our package bcdata (https://cran.r-project.org/package=bcdata) retrieves public data from British Columbia data services, so the examples are dependent on successfully hitting web resources. When we initially submitted to CRAN the examples were wrapped in \dontrun{}, but we were asked to change them to use \donttest{}. Now, CRAN is running checks with _R_CHECK_DONTTEST_EXAMPLES_=true, and one of the examples has failed due to a changed web resource. I got an email asking me to fix it and reminding me of the CRAN policy: 'Packages which use Internet resources should fail gracefully with an informative message if the resource is not available or has changed (and not give a check warning nor error).' This needs correction whether or not the resource recovers. I would prefer that in normal use, the function calling the internet resources fails with an error so that the user is properly alerted. I also agree that these failures shouldn't cause a check error. To demonstrate the use of these functions in examples, is there a preferred method of ensuring that failures will not cause a check warning or error? Should I try to revert back to using \dontrun{} and plead my case? Wrap in `if interactive())`? Thanks very much, Andy Teucher
[R-pkg-devel] Usage of internet resources in examples
10 messages · Andy Teucher, Gábor Csárdi, Ben Bolker +2 more
Depending on what you want to do, if (interactive()) or try() (in the examples, not in the function) or if (testthat::skip_on_cran()) all seem like reasonable solutions.
On 10/20/20 4:28 PM, Andy Teucher wrote:
Our package bcdata (https://cran.r-project.org/package=bcdata) retrieves public data from British Columbia data services, so the examples are dependent on successfully hitting web resources. When we initially submitted to CRAN the examples were wrapped in \dontrun{}, but we were asked to change them to use \donttest{}. Now, CRAN is running checks with _R_CHECK_DONTTEST_EXAMPLES_=true, and one of the examples has failed due to a changed web resource. I got an email asking me to fix it and reminding me of the CRAN policy: 'Packages which use Internet resources should fail gracefully with an informative message if the resource is not available or has changed (and not give a check warning nor error).' This needs correction whether or not the resource recovers. I would prefer that in normal use, the function calling the internet resources fails with an error so that the user is properly alerted. I also agree that these failures shouldn't cause a check error. To demonstrate the use of these functions in examples, is there a preferred method of ensuring that failures will not cause a check warning or error? Should I try to revert back to using \dontrun{} and plead my case? Wrap in `if interactive())`? Thanks very much, Andy Teucher
______________________________________________ R-package-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
On Tue, Oct 20, 2020 at 9:45 PM Ben Bolker <bbolker at gmail.com> wrote:
[...]
if (testthat::skip_on_cran()) all seem like reasonable solutions.
I don't think you can use `testthat::skip_on_cran()` for this, it does not return a logical flag: ? isTRUE(testthat::skip_on_cran()) Error: Reason: On CRAN It only works in testthat tests. Gabor [...]
On 10/20/20 4:51 PM, G?bor Cs?rdi wrote:
On Tue, Oct 20, 2020 at 9:45 PM Ben Bolker <bbolker at gmail.com> wrote: [...]
if (testthat::skip_on_cran()) all seem like reasonable solutions.
I don't think you can use `testthat::skip_on_cran()` for this, it does not return a logical flag: ? isTRUE(testthat::skip_on_cran()) Error: Reason: On CRAN It only works in testthat tests. Gabor [...]
oh OK, sorry. The guts of the function are (in testthat:::on_cran(),
which can't be used because it's not exported)
!identical(Sys.getenv("NOT_CRAN"), "true")
On Tue, Oct 20, 2020 at 10:07 PM Ben Bolker <bbolker at gmail.com> wrote:
On 10/20/20 4:51 PM, G?bor Cs?rdi wrote:
On Tue, Oct 20, 2020 at 9:45 PM Ben Bolker <bbolker at gmail.com> wrote: [...]
if (testthat::skip_on_cran()) all seem like reasonable solutions.
I don't think you can use `testthat::skip_on_cran()` for this, it does not return a logical flag: ? isTRUE(testthat::skip_on_cran()) Error: Reason: On CRAN It only works in testthat tests. Gabor [...]
oh OK, sorry. The guts of the function are (in testthat:::on_cran(),
which can't be used because it's not exported)
!identical(Sys.getenv("NOT_CRAN"), "true")
It is not really worth exporting this, because it just checks for an environment variable, that's all it does. So all users would need to set this environment variable to be able to run the examples. Which does not seem like a great solution. G.
On 10/20/20 5:10 PM, G?bor Cs?rdi wrote:
On Tue, Oct 20, 2020 at 10:07 PM Ben Bolker <bbolker at gmail.com> wrote:
On 10/20/20 4:51 PM, G?bor Cs?rdi wrote:
On Tue, Oct 20, 2020 at 9:45 PM Ben Bolker <bbolker at gmail.com> wrote: [...]
if (testthat::skip_on_cran()) all seem like reasonable solutions.
I don't think you can use `testthat::skip_on_cran()` for this, it does not return a logical flag: ? isTRUE(testthat::skip_on_cran()) Error: Reason: On CRAN It only works in testthat tests. Gabor [...]
oh OK, sorry. The guts of the function are (in testthat:::on_cran(),
which can't be used because it's not exported)
!identical(Sys.getenv("NOT_CRAN"), "true")
It is not really worth exporting this, because it just checks for an environment variable, that's all it does. So all users would need to set this environment variable to be able to run the examples. Which does not seem like a great solution. G.
OK, fair enough. Ben B.
I set \dontrun{} on (most of) the examples in my Azure-related packages, for this very reason. They assume the user has an Azure subscription, and in any case, you don't want to be running code that could potentially cost lots of money just for kicks. The packages in question are basically all those on CRAN that start with "Azure".
I cleared this with Uwe Ligges some time back; although whenever I submit a new package, I tend to get emails from whoever did the review that \donttest() should be removed. I've been able to resolve this by replying and cc'ing Uwe, mentioning that it was approved previously.
-----Original Message-----
From: R-package-devel <r-package-devel-bounces at r-project.org> On Behalf Of Andy Teucher
Sent: Wednesday, 21 October 2020 7:29 AM
To: r-package-devel at r-project.org
Subject: [EXTERNAL] [R-pkg-devel] Usage of internet resources in examples
Our package bcdata (https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcran.r-project.org%2Fpackage%3Dbcdata&data=04%7C01%7Chongooi%40microsoft.com%7C3a5d0a6f36074ebfde8408d87536d145%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637388226515752313%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=xxnnjkhi%2B40WOm6S9VBrAB8M4IPePX0MrvN3oosqYnA%3D&reserved=0) retrieves public data from British Columbia data services, so the examples are dependent on successfully hitting web resources. When we initially submitted to CRAN the examples were wrapped in \dontrun{}, but we were asked to change them to use \donttest{}.
Now, CRAN is running checks with _R_CHECK_DONTTEST_EXAMPLES_=true, and one of the examples has failed due to a changed web resource. I got an email asking me to fix it and reminding me of the CRAN policy:
'Packages which use Internet resources should fail gracefully with an informative message
if the resource is not available or has changed (and not give a check warning nor error).'
This needs correction whether or not the resource recovers.
I would prefer that in normal use, the function calling the internet resources fails with an error so that the user is properly alerted. I also agree that these failures shouldn't cause a check error. To demonstrate the use of these functions in examples, is there a preferred method of ensuring that failures will not cause a check warning or error? Should I try to revert back to using \dontrun{} and plead my case? Wrap in `if interactive())`?
Thanks very much,
Andy Teucher
______________________________________________
R-package-devel at r-project.org mailing list
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstat.ethz.ch%2Fmailman%2Flistinfo%2Fr-package-devel&data=04%7C01%7Chongooi%40microsoft.com%7C3a5d0a6f36074ebfde8408d87536d145%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637388226515752313%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=qzv9k50S770BrVY%2BAAsDE8i0w8C3zW3reZtxZOJdW9E%3D&reserved=0
Thanks everyone for the responses. I?m not sure why I didn?t think of wrapping the examples in try(), but I think that might be the best solution - allows the examples to be run, but won?t throw errors in R CMD check. Hong Ooi - that?s a good point though I think your example does fall more squarely into the ?use dontrun? category as they require a subscription/API key. Our examples don?t, but they can fail if the internet connection fails, the resource url changes, or the server is down. Cheers, Andy
On Oct 20, 2020, at 2:42 PM, Hong Ooi <hongooi at microsoft.com> wrote:
I set \dontrun{} on (most of) the examples in my Azure-related packages, for this very reason. They assume the user has an Azure subscription, and in any case, you don't want to be running code that could potentially cost lots of money just for kicks. The packages in question are basically all those on CRAN that start with "Azure".
I cleared this with Uwe Ligges some time back; although whenever I submit a new package, I tend to get emails from whoever did the review that \donttest() should be removed. I've been able to resolve this by replying and cc'ing Uwe, mentioning that it was approved previously.
-----Original Message-----
From: R-package-devel <r-package-devel-bounces at r-project.org <mailto:r-package-devel-bounces at r-project.org>> On Behalf Of Andy Teucher
Sent: Wednesday, 21 October 2020 7:29 AM
To: r-package-devel at r-project.org <mailto:r-package-devel at r-project.org>
Subject: [EXTERNAL] [R-pkg-devel] Usage of internet resources in examples
Our package bcdata (https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcran.r-project.org%2Fpackage%3Dbcdata&data=04%7C01%7Chongooi%40microsoft.com%7C3a5d0a6f36074ebfde8408d87536d145%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637388226515752313%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=xxnnjkhi%2B40WOm6S9VBrAB8M4IPePX0MrvN3oosqYnA%3D&reserved=0 <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcran.r-project.org%2Fpackage%3Dbcdata&data=04%7C01%7Chongooi%40microsoft.com%7C3a5d0a6f36074ebfde8408d87536d145%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637388226515752313%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=xxnnjkhi%2B40WOm6S9VBrAB8M4IPePX0MrvN3oosqYnA%3D&reserved=0>) retrieves public data from British Columbia data services, so the examples are dependent on successfully hitting web resources. When we initially submitted to CRAN the examples were wrapped in \dontrun{}, but we were asked to change them to use \donttest{}.
Now, CRAN is running checks with _R_CHECK_DONTTEST_EXAMPLES_=true, and one of the examples has failed due to a changed web resource. I got an email asking me to fix it and reminding me of the CRAN policy:
'Packages which use Internet resources should fail gracefully with an informative message
if the resource is not available or has changed (and not give a check warning nor error).'
This needs correction whether or not the resource recovers.
I would prefer that in normal use, the function calling the internet resources fails with an error so that the user is properly alerted. I also agree that these failures shouldn't cause a check error. To demonstrate the use of these functions in examples, is there a preferred method of ensuring that failures will not cause a check warning or error? Should I try to revert back to using \dontrun{} and plead my case? Wrap in `if interactive())`?
Thanks very much,
Andy Teucher
______________________________________________ R-package-devel at r-project.org <mailto:R-package-devel at r-project.org> mailing list https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstat.ethz.ch%2Fmailman%2Flistinfo%2Fr-package-devel&data=04%7C01%7Chongooi%40microsoft.com%7C3a5d0a6f36074ebfde8408d87536d145%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637388226515752313%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=qzv9k50S770BrVY%2BAAsDE8i0w8C3zW3reZtxZOJdW9E%3D&reserved=0 <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstat.ethz.ch%2Fmailman%2Flistinfo%2Fr-package-devel&data=04%7C01%7Chongooi%40microsoft.com%7C3a5d0a6f36074ebfde8408d87536d145%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637388226515752313%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=qzv9k50S770BrVY%2BAAsDE8i0w8C3zW3reZtxZOJdW9E%3D&reserved=0>
Ben Bolker
on Tue, 20 Oct 2020 17:07:10 -0400 writes:
> On 10/20/20 4:51 PM, G?bor Cs?rdi wrote:
>> On Tue, Oct 20, 2020 at 9:45 PM Ben Bolker <bbolker at gmail.com> wrote:
>> [...]
>>> if (testthat::skip_on_cran())
>>>
>>> all seem like reasonable solutions.
>>
>> I don't think you can use `testthat::skip_on_cran()` for this, it does
>> not return a logical flag:
>>
>> ? isTRUE(testthat::skip_on_cran())
>> Error: Reason: On CRAN
>>
>> It only works in testthat tests.
>>
>> Gabor
>>
>> [...]
>>
> oh OK, sorry. The guts of the function are (in testthat:::on_cran(),
> which can't be used because it's not exported)
> !identical(Sys.getenv("NOT_CRAN"), "true")
I still disagree with that part of your recommendations, for this reason:
I think a "perfect" solution would keep the example(s) running
fine when the user calls
example(<topic>)
(hence it is very good recommendation to *not* misuse \dontrun{.} here)
*and* have the example(.) call *not* stop on error, but (say)
produce a warning in the case the internet connection isn't
stable at the time the user calls example(.),
e.g., myself in a train ride in the Swiss alps ..
Hence, your (Ben's) first two suggestions make more sense to me,
and I'd clearly favor
try(.)
or possibly slightly more sophsticated versions such as
if(inherits(try({
.......
}), "try-error")) warning(....)
Or then your 2nd suggestion, if(interactive())
in the case (as it seems here) where the package author
really wants the user to get an error when the internet
connection fails.
Martin
Thanks Martin, I agree that the example should run for a user when calling example() - hence I am going with a simple try() around the function calls in the examples. I agree your more sophisticated version has better behaviour, but I don?t want to clutter the examples with code that might just confuse a user. Andy
On Oct 21, 2020, at 12:08 AM, Martin Maechler <maechler at stat.math.ethz.ch> wrote:
Ben Bolker on Tue, 20 Oct 2020 17:07:10 -0400 writes:
On 10/20/20 4:51 PM, G?bor Cs?rdi wrote:
On Tue, Oct 20, 2020 at 9:45 PM Ben Bolker <bbolker at gmail.com> wrote: [...]
if (testthat::skip_on_cran()) all seem like reasonable solutions.
I don't think you can use `testthat::skip_on_cran()` for this, it does not return a logical flag: ? isTRUE(testthat::skip_on_cran()) Error: Reason: On CRAN It only works in testthat tests. Gabor [...]
oh OK, sorry. The guts of the function are (in testthat:::on_cran(), which can't be used because it's not exported)
!identical(Sys.getenv("NOT_CRAN"), "true")
I still disagree with that part of your recommendations, for this reason:
I think a "perfect" solution would keep the example(s) running
fine when the user calls
example(<topic>)
(hence it is very good recommendation to *not* misuse \dontrun{.} here)
*and* have the example(.) call *not* stop on error, but (say)
produce a warning in the case the internet connection isn't
stable at the time the user calls example(.),
e.g., myself in a train ride in the Swiss alps ..
Hence, your (Ben's) first two suggestions make more sense to me,
and I'd clearly favor
try(.)
or possibly slightly more sophsticated versions such as
if(inherits(try({
.......
}), "try-error")) warning(....)
Or then your 2nd suggestion, if(interactive())
in the case (as it seems here) where the package author
really wants the user to get an error when the internet
connection fails.
Martin
______________________________________________ R-package-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel