R 3.1.0: 'R CMD Sweave' deletes non tex files created upon batch mode exit

Marc Schwartz <marc_schwartz at me.com>
   on Sun, 13 Apr 2014 10:22:55 -0500 writes:

[on the R-SIG-Mac  mailing list] :	  

Hi all,
With R version 3.1.0 on OSX, using either the Snow Leopard or the Mavericks binary installation on a Mac with fully updated Mavericks, there has been a change in behavior since 3.0.3.

I've just written to R-core about this:

   R CMD Sweave

has *serious* problems also in the very simple case when it
should produce figures: They are not available after the
completion of R CMD Sweave,
confirming your  'deletes non tex files created upon batch mode exit'

So this is not related to OSX only, but also a big problem 
at least on other *nix descendent platforms, such as Linux.

In short, failure (no graphic produce in the example below) by

  R CMD Sweave foo.Rnw

but all is fine with using Sweave(), as you Marc, noted, and hence with  

  Rscript -e 'Sweave("foo.Rnw")'

... and to answer your question:  
No this was not intended and is probably one of the bigger / 
most embarrassing bugs in a newly released version of R
in my view: 

Basically  'R CMD Sweave' is partly broken in R 3.1.0.
Yes, this should never have happened.

I had to partially revert the R 3.1.0 installation here (our
statistics dept), by making R 3.0.3 the default 'R'
for now, as we are relying on 'R CMD Sweave ..' in many places.

Personally I've never noticed the problem, as I seem to always
Sweave from ESS (Emacs speaks statistics) which calls  Sweave()
in R, and that works fine, also in  R 3.1.0, as you've already
noted 

Martin Maechler, ETH Zurich

%------------------------------------------------------------
\documentclass[12pt]{article}
\usepackage{Sweave}
\begin{document}  Just a simple graphic 

<<qqnorm, fig=TRUE>>= 
qqnorm(rnorm(20))
@ 

and that's all, folks!
\end{document}
%------------------------------------------------------------

I have a master .Rnw file which runs a series of outputs from multiple R code files, each called in BATCH mode using system() from within the master .Rnw file. The output of the R code files go to separate text files in order to catch some of the function call output that would not otherwise be included in the resultant .tex file due to output redirection.

Those text files are then included in the resultant .tex file using, for example:

\lstinputlisting[caption={}]{test.out}

directives which are included in the .Rnw source file.

A simple example to replicate the observed behaviors.

The test.Rnw file content:

%% R CMD Sweave test.Rnw
<<results=tex,echo=false>>=
system("R CMD BATCH test.R test.out")
@ 

The test.R file content:

options(echo = FALSE)
options(useFancyQuotes = FALSE)
installed.packages()

On version 3.0.3, the file test.out is created, along with test.tex. test.out contains the output of installed.packages(). I did not include the aforementioned listing directive in test.Rnw here for simplicity.

On version 3.1.0, the file test.out is created, but when the R CMD Sweave command exits and returns to the CLI in the console, test.out is deleted, presumably as part of a post R batch session clean up process. The file test.tex is retained.

I uninstalled 3.1.0 and reinstalled 3.0.3 and observed the prior behavior. I then tried clean installs of both the Snow Leopard and Mavericks 3.1.0 binaries, with the new behavior observed in both cases.

In reading the NEWS file for 3.1.0, there are multiple references to Sweave, but there is nothing explicit about this new behavior. 

I should note that when the .Rnw file is run from within an R 3.1.0 interactive session using:

Sweave("test.Rnw")

the test.out file is created and not deleted upon the function exit, or when exiting the R session back to the console. 

Thus, this new behavior seems to be limited to running Sweave from the CLI using R CMD. It is not clear to me if this new behavior is by design or perhaps an unintended consequence of changes elsewhere in Sweave processing or in the handling of R in BATCH mode.

When watching the folder where the file output activity takes place, I note that a file .build.timestamp is created and then deleted, which leads me to believe that the new functions fileSnapshot() and changedFiles() are being used in the course of processing R in BATCH mode. If that is correct and there is a clean up step that is occurring upon BATCH mode exit, which deletes all files and folders other than .tex files, is it possible that this process is being overly aggressive? Or if not, is there a way to not have these files deleted or have a protected folder where these files can be retained?

Thanks for any insights.

Regards,

Marc Schwartz