Skip to content

[R-pkg-devel] Run-away R CMD check ?

8 messages · Duncan Murdoch, Paul Gilbert, Dirk Eddelbuettel +1 more

Original
Dirk Eddelbuettel
# 
For Rcpp and friends, I frequently run 'R CMD check' in "batch mode" over a
large set of packages.  That works fine, apart from taking long :)

I am using the exact same script (which is home-grown and could do with a
revision, but is public at [1] -- basically just a big loop over a set of
packages), and the exact same ~/.R/check.Renviron.

Yet on the (bigger, better, newerm ...) machine at work I have three packages
which get into a loop and never finish R CMD check.

Any idea why / how and what to do?  Besides skipping these three packages
which I'll do next?

The only commonality is that all three use testthat, but so do many other
packages that test fine.  

Dirk

[1] https://github.com/RcppCore/rcpp-logs/blob/master/scripts/runRcppDepends.r
Duncan Murdoch
#
On 12/12/2015 9:30 AM, Dirk Eddelbuettel wrote:
I'd put a time limit on the process so it fails eventually, and then see 
what the log shows just before the failure.  That is, I would if I could 
remember how to do that.

Duncan Murdoch
Dirk Eddelbuettel
#
On 12 December 2015 at 11:00, Duncan Murdoch wrote:
| On 12/12/2015 9:30 AM, Dirk Eddelbuettel wrote:
| >
| > For Rcpp and friends, I frequently run 'R CMD check' in "batch mode" over a
| > large set of packages.  That works fine, apart from taking long :)
| >
| > I am using the exact same script (which is home-grown and could do with a
| > revision, but is public at [1] -- basically just a big loop over a set of
| > packages), and the exact same ~/.R/check.Renviron.
| >
| > Yet on the (bigger, better, newerm ...) machine at work I have three packages
| > which get into a loop and never finish R CMD check.
| >
| > Any idea why / how and what to do?  Besides skipping these three packages
| > which I'll do next?
| >
| > The only commonality is that all three use testthat, but so do many other
| > packages that test fine.
| >
| > Dirk
| >
| > [1] https://github.com/RcppCore/rcpp-logs/blob/master/scripts/runRcppDepends.r
| >
| 
| I'd put a time limit on the process so it fails eventually, and then see 
| what the log shows just before the failure.  That is, I would if I could 
| remember how to do that.

Yes. I thought about watcher processes too. But I haven't come up with
something simple enough to implement.

So for now I will just skip those three bad boys. Two of them, as I recall,
are merely unreasonable and finish after a while.  The other way ran all night.

I'd love to know how CRAN deals with that, or understand better why it
doesn't bite me on the server at home.

Dirk
Paul Gilbert
#
On 12/12/2015 11:00 AM, Duncan Murdoch wrote:
Two possibilities occur to me.
1/ Since I am just looking for a way to eliminate "No repository set..." 
and see you had the same concern once, if you actually solved that you 
may be getting into the "...cyclic dependency check" which notes in the 
code suggest may give an endless loop. I've skipped several details I 
don't understand.

2/ You can get a lock condition with make -j 2. For example, I have a 
test that can clobber a test db that another test is using, so I have to 
be carefully not to run these in parallel. This is easy to check, if you 
try without parallel make and don't have a problem then it is likely the 
issue.

Paul Gilbert
Dirk Eddelbuettel
#
On 12 December 2015 at 11:36, Paul Gilbert wrote:
| 
|
| On 12/12/2015 11:00 AM, Duncan Murdoch wrote:
| > On 12/12/2015 9:30 AM, Dirk Eddelbuettel wrote:
| >>
| >> For Rcpp and friends, I frequently run 'R CMD check' in "batch mode"
| >> over a
| >> large set of packages.  That works fine, apart from taking long :)
| >>
| >> I am using the exact same script (which is home-grown and could do with a
| >> revision, but is public at [1] -- basically just a big loop over a set of
| >> packages), and the exact same ~/.R/check.Renviron.
| >>
| >> Yet on the (bigger, better, newerm ...) machine at work I have three
| >> packages
| >> which get into a loop and never finish R CMD check.
| >>
| >> Any idea why / how and what to do?  Besides skipping these three packages
| >> which I'll do next?
| 
| Two possibilities occur to me.
| 1/ Since I am just looking for a way to eliminate "No repository set..." 
| and see you had the same concern once, if you actually solved that you 
| may be getting into the "...cyclic dependency check" which notes in the 
| code suggest may give an endless loop. I've skipped several details I 
| don't understand.

No, it's not that. These build fine; they 'merely' go on too long when
running the tests.
 
| 2/ You can get a lock condition with make -j 2. For example, I have a 
| test that can clobber a test db that another test is using, so I have to 
| be carefully not to run these in parallel. This is easy to check, if you 
| try without parallel make and don't have a problem then it is likely the 
| issue.

That is a really good suggestion.  I have so little ram on the older server
here at home that don't usually set a parallel make.   At work I do, and that
may well be the difference.  Will test and report back.

Thanks!

Dirk
Uwe Ligges
#
On 12.12.2015 17:34, Dirk Eddelbuettel wrote:
CRAN has different approaches.

Under Windows, I typically have an R process that controls the check 
process and writes out the process id and timestamp and then starts R 
CMD check. Another watcher simply kills the process of a given process 
ID including all childs if they are beyond some timeout threshold.

Best,
Uwe
Dirk Eddelbuettel
#
On 12 December 2015 at 18:57, Uwe Ligges wrote:
| CRAN has different approaches.
| 
| Under Windows, I typically have an R process that controls the check 
| process and writes out the process id and timestamp and then starts R 
| CMD check. Another watcher simply kills the process of a given process 
| ID including all childs if they are beyond some timeout threshold.

I was leaning towards something like that. We'll see what I end up doing...

Dirk
Dirk Eddelbuettel
#
On 12 December 2015 at 12:10, Dirk Eddelbuettel wrote:
|
| On 12 December 2015 at 18:57, Uwe Ligges wrote:
| | CRAN has different approaches.
| | 
| | Under Windows, I typically have an R process that controls the check 
| | process and writes out the process id and timestamp and then starts R 
| | CMD check. Another watcher simply kills the process of a given process 
| | ID including all childs if they are beyond some timeout threshold.
| 
| I was leaning towards something like that. We'll see what I end up doing...

I am also going to keep closer tabs on how long each test takes -- some
packages are clearly more expensive than others.

Dirk