[R-pkg-devel] CRAN submission error when running tests in testthat

On Thu, 25 Nov 2021 12:33:01 +0100

Are you sure about this? I don't think testthat hides any output. R
CMD check on the other hand AFAICT only shows the last 13 lines by
default. See the _R_CHECK_TESTS_NLINES_ environment variable in
https://cran.r-project.org/doc/manuals/r-devel/R-ints.html

Indeed you're right, R CMD check truncating the output is the main
problem here. With a full traceback, it would at least be possible to
see which dependency crashes R on CRAN Macs. I should have been more
clear in my e-mail.

By "testthat hiding output" I mean that there's no *.Rout[.fail] files
for every file in tests/testthat/*.R with line-by-line output of the
test code. In cases when R itself crashes in the middle of the test run,
these *.Rout.fail files could be useful to find out which line caused
the crash. With testthat, crashing the R process during tests results
in a single testthat.Rout.fail:

library(testthat)
library(package_name)

...

test_check("package_name")

 *** caught segfault ***
address (nil), cause 'memory not mapped'

Traceback:
 1: .C("foo")
 2: eval(code, test_env)

...

23: test_dir("testthat", package = package, reporter = reporter,     ..., load_package = "installed")
24: test_check("package_name")
An irrecoverable exception occurred. R is aborting now ...

Here, the traceback makes the answer obvious, but it's not always so.
With serious heap corruption, it's possible to get just:

test_check("package_name")

free(): invalid pointer
Aborted

With no indication of the call or the file causing the crash.

You are right that this is not the main problem here: if the test
output wasn't truncated by R CMD check, we would know the origin of the
crash, even without testthat.

Best regards,
Ivan