R CMD check recursive copy of tests/
Hi.
On 8/30/07, Martin Maechler <maechler at stat.math.ethz.ch> wrote:
"HenrikB" == Henrik Bengtsson <hb at stat.berkeley.edu>
on Wed, 29 Aug 2007 15:11:17 -0700 writes:
>> From NEWS of R v2.6.0 devel:
HenrikB> o R CMD check now does a recursive copy on the
HenrikB> 'tests' directory.
HenrikB> However, R CMD check does not run *.R scripts in
HenrikB> such subdirectories (as I thought/hoped for),
a pretty bold hope ;-)
HenrikB> only those directly under tests/, This may or may not be
HenrikB> intentional.
intentional I'd say: I did not implement it, but it seems much
more logical to keep the previous rule: All *.R files in
./tests/ are run <period>
Subdirectories can be useful for organization, notably storing
test data. I don't think it's a good idea to use so very many test files
that you need subdirectories, unless maybe you are thinking
about unit tests; and then, see below.
Examples of subdirectories (some overlapping) are: units/ - tests of minimal code modules integration/ - tests of integrating the above units system/ - "real-world" scenarios/use cases requirements/ - every requirement should have at least on test. bugs/ - every bug fix should come with a new test. regression/ - every update should have a regression test to validate backward compatibility etc. robustness/ - Testing the robustness of estimators against outliers as well as extreme parameter settings. validation/ - validation of numeric results compared with alternative implementations or summaries. benchmarking/ - actually more measuring time, but can involve validation that a method is faster than an alternative. crossplatform/ - validate correctness across platforms. torture/ - pushing the limits.
HenrikB> If true, maybe the above should be clarified as:
HenrikB> o R CMD check now does a recursive copy on the
HenrikB> 'tests' directory for the purpose of provided data
HenrikB> files for input. Test scripts still has to be
HenrikB> directly under tests/ to be run.
HenrikB> Just a comment
Thanks for the comment.
Many other people have complained that 'NEWS' is too verbose
already, and hence too much to be read by an average useR, even programmeR.
The point about 'NEWS' is to mention things that are *changed*.
Now since the behavior did *not* change, we definitely should
not make NEWS longer just to say that.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
A note about ./tests/ , directories, etc :
At the useR!2007 meeting three weeks ago, there was an
interesting discussion --- I think after the "Regulated
Environments / R Certification" session {{and those interested
there should look at http://www.r-project.org/certification.html}} ---
I ran in to that last week; I think that is a great and important initiative. Now I hear that people use R for development and then reimplement in SAS to please FDA.
on the broader topic of 'R validation' and testing etc,
where one of my short contributions was to acknowledge that
"yes, it would be great if people wrote much more
regression/unit test code, and I would welcome if these were
collected, and put into daily testing state etc etc"
*Contrary* to the <pkg>/tests/ concept,
one thing I think we (R developers at large) should consider and
often favor, is the idea that you want to run the tests also at
run time and not just install time
{{and part of that is also desirable for "base R"}}
About a week agao, I have re-organized the
RUnit-based unit regression tests for two Rmetrics R packages to
achieve the following goals :
1) tests should be auto-run via 'R CMD check <pkg>'
2) tests should be easily re-runnable by a user who only
got the *binary* package {e.g. a typical Windows user}
3) tests should also be easily runnable by the package developer
without need to run the full R CMD check.
where "1)" and "2)" where really primary goals.
I've basically started from what is written (I think mainly by
Tony Plate and Gregor Gorjanc) in the R Wiki section
http://wiki.r-project.org/rwiki/doku.php?id=developers:runit
and Gregor Gojanc's example in CRAN package 'gdata'
but then really also found I needed to change things a bit to
make '2)' more convenient.
The basic idea is:
./tests/doRunit.R is a setup file {which will not be installed}
./inst/unitTests/ contains all the meat {and *is* installed}
with
./inst/unitTests/runTests.R calls all the other unit tests
./inst/unitTests/runit*.R
where the idea is the user of the *installed* package
can easily run the tests again,
hereby **validating** her installation of the package:
E.g., for the not-yet released version of fCalendar,
source(system.file("unitTests/runTests.R", package = "fCalendar"))
RUnit 0.4.17 loaded.
Now have RUnit Test Suite 'testSuite' for package 'fCalendar' :
......
Consider doing
tests <- runTestSuite(testSuite)
and later
printTextProtocol(tests)
I think these are all great ideas/suggests/implementation. As important as design and implementation, testing (and bug reporting) deserves a higher status. It often the case that it is not feasible to have all tests ran by R CMD check. For instance, I have system tests that take literally hours or even days to run. I also know that because of their integrated build/check system, Bioconductor have a rule of thumb that it shouldn't take more than 5 minutes for R CMD check to finish [http://wiki.fhcrc.org/bioc/Package_Guidelines#time-requirements]. Then having test cases installed allows you to still include such tests and run them at other times (run time). I would like to add that it can also be useful to have the option tp ask an end-user that reports bugs/problems to run some standard tests (that might not have been ran by R CMD check). Cheers Henrik
-- -- -- -- -- The development version is available via subversion (svn) or also simple web tools from https://svn.r-project.org/Rmetrics/trunk/fCalendar/ Martin Maechler, ETH Zurich