Skip to content

Checking package help file examples

3 messages · Bill Dunlap, Arne Henningsen

Original
Bill Dunlap
# 
I am trying to figure out the sanctioned way for
'R CMD check pkg' to make sure that the examples
in help files give the expected results.

Writing R Extensions says that check runs the help
file examples (which INSTALL extracts from pkg/man/*.Rd
and puts into files in pkg/R-ex).  It looks like check
concatenates all the example files, pkg/R-ex/*.R, into one
file, pkg/pkg-Ex.R, which it runs to produce pkg/pkg-Ex.Rout.
If there is an error running these files then check aborts.

I don't seen anything in Writing R Extensions about
checking that correct results were produced.  However,
in R 2.7.0, check will compare pkg/pkg-Ex.Rout to
pkg/tests/Examples/pkg/pkg-Ex.Rout.save, if the latter
file is in your package source, and report any differences.

I don't see any mention of this and was wondering if this
method should be recommended or if there was a preferred
way to check help file examples for correctness.  (It would
be nicer if the example() function could also check for
correctness, and the above method doesn't allow for that.)

----------------------------------------------------------------------------
Bill Dunlap
Insightful Corporation
bill at insightful dot com
360-428-8146

 "All statements in this message represent the opinions of the author and do
 not necessarily reflect Insightful Corporation policy or position."
Arne Henningsen
#
On Friday 18 July 2008 02:19:14, Bill Dunlap wrote:
I use *.R and corresponding *.Rout.save files in the subdirectory "tests" for 
checking packages (see e.g., packages systemfit, micEcon, 
sampleSelection, ...). I think that it is better to separate examples from 
testing, because IMHO there should be a few simple examples but tests should 
contain many different function calls with (partly) unusual arguments to 
check (nearly) the entire functionality of each function.

See also the manual "Writing R Extensions"
http://cran.r-project.org/doc/manuals/R-exts.html#Package-subdirectories
======================================
Subdirectory tests is for additional package-specific test code, similar to 
the specific tests that come with the R distribution. Test code can either be 
provided directly in a .R file, or via a .Rin file containing code which in 
turn creates the corresponding .R file (e.g., by collecting all function 
objects in the package and then calling them with the strangest arguments). 
The results of running a .R file are written to a .Rout file. If there is a 
corresponding .Rout.save file, these two are compared, with differences being 
reported but not causing an error. The directory tests is copied to the check 
area, and the tests are run with the copy as the working directory and with 
R_LIBS set to ensure that the copy of the package installed during testing 
will be found by library(pkg_name). 
======================================

Best wishes,
Arne
Bill Dunlap
#
On Fri, 18 Jul 2008, Arne Henningsen wrote:

            

      
      
      
    

        
oops: pkg/tests/Examples/pkg-Ex.Rout.save

            

      
      
      
    

        
My question was about the undocumented code in R_HOME/bin/check
that compares the output from running the help file examples,
pkg.Rcheck/pkg-Ex.Rout, to the file pkg/tests/Examples/pkg-Ex.Rout.save,
if the latter file is in the package source.

        $log->checking("examples");
        ... ${pkgname}-Ex.R was created by massage-Examples.pl ...
        ... run "R < ${pkgname}-Ex.R > ${pkgname}-Ex.Rout" ...
        ## Try to compare results from running the examples to a saved
        ## previous version.
        my $Rex_out_save =
            &file_path($pkgdir, "tests", "Examples",
                       "${pkgname}-Ex.Rout.save");
        if(-f $Rex_out_save) {
            $log->checking("differences from '${pkgname}-Ex.Rout' to '${pkgname}-Ex.Rout.save'");
            $cmd = join(" ",
                        (${R::Vars::R_CMD},
                         "Rdiff",
                         &shell_quote_file_path("${pkgname}-Ex.Rout"),
                         &shell_quote_file_path("${Rex_out_save}"),
                         "0"));
            if(R_system($cmd)) {
                $log->error();
                $log->print("Comparison failed.\n");
                exit(1);
            }
            $log->result("OK");
        }
    }

A package writer noticed the pkg-Ex.Rout file created by
check and wondered if this was ever compared to a Rout.save
file as is done with other Rout files and I noticed the
above perl code that did so.  I didn't see any CRAN packages
that had a tests/Examples/pkg-Ex.Rout.save file, but some
have a pkg-Ex.Rout.save in a different place.

Should we document the current behavior (should I use this
scheme in Splus packages)?  Should we get rid of the code
or put in comments that it should not be used?

If I were writing this from scratch, I would not put all
the example output into one Rout or Rout.save file, but
put use one Rout file per help file.  Then the results
would not depend on the local collating order.

I agree that detailed tests belong in tests, but since check
runs the help file example to make sure they don't throw
errors, it seems reasonable to also check that they give
the expected results.

            

      
      
      
    

        
----------------------------------------------------------------------------
Bill Dunlap
Insightful Corporation
bill at insightful dot com

 "All statements in this message represent the opinions of the author and do
 not necessarily reflect Insightful Corporation policy or position."