CRAN test / avoidance

( subject changed from Re: [Rd] R-devel Digest, Vol 115, Issue 18 )

I have the impression from this, and previous discussions on the 
subject, that package developers and CRAN maintainers are talking at 
cross-purposes. Many package maintainers are thinking that they should 
be responsible for choosing which tests are run and which are not run by 
CRAN, whereas CRAN maintainers may want to run all possible tests 
sometimes, or a trimmed down set when time constraints demand this. With 
good reason, CRAN may want to run all possible tests sometimes. There 
are too many packages on CRAN that remain there because they don't have 
any testing or vignettes, and very few examples. Encouraging more of 
that is a bad thing.

If I understand correctly, the --as-cran option was introduced to help 
developers specify options that CRAN uses, so they would find problems 
that CRAN would notice, and correct before submitting. The Rd 
discussions of this have morphed into a discussion of how package 
developers can use --as-cran to control which tests are run by CRAN.

I tend to be more sympathetic with what I call the CRAN maintainer view 
above, even though I am a package developer. I think packages should 
have extensive testing and that all the tests should go in the source 
package on CRAN, so the testing is available for CRAN and everyone else. 
(Although, it is sometimes not clear if CRAN maintainers like me doing 
this, because they are torn between time demands and maintaining quality 
- that is part of the confusion.)

The question becomes: how does information get passed along to indicate 
things that may take a long time to run. The discussion so far has 
focused on developers setting, or using, some flags to indicate tests 
and examples that take a long time. Another option would be to have the 
check/build process generate a file with information about the time it 
took to run tests, vignettes, and examples, probably with some 
information about the speed of the machine it was run on. Then CRAN and 
anyone else that wants to run tests can take this information into 
consideration.

Paul

In general, as a package user, I don't want people to be able to
suppress checks on CRAN.  I want things fixed.

So I am pretty sure there won't ever be a reliable "CRAN-detector" put
into R.  It would devalue the brand.

Duncan Murdoch

My problem is that CRAN demands that I suppress a large fraction of my
checks, in order to fit within time constraints.  This leaves me with 3
choices.

1. Add lines to my code that tries to guess if CRAN is invoker.  A cat
and mouse game per your desire above.

2. Remove large portions of my test suite.  I consider the survival
package to be one of the pre-eminent current code sets in the world
precisely because of the extensive validations, this action would change
it to a second class citizen.

3. Add a magic environment variable to my local world, only do the full
tests if it is present, and make the dumbed down version the default.
Others who want to run the full set are then SOL, which I very much
don't like.

I agree that CRAN avoidence, other than the time constraint, should be
verboten.  But I don't think that security through obscurity is the
answer.  And note that under scenario 3, which is essentially what is
currently being forced on us, I can do such micshief as easily as under
number 1.

Terry Therneau

______________________________________________
R-devel at r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel