Skip to content

R 3.1.0: 'R CMD Sweave' deletes non tex files created upon batch mode exit

16 messages · Marc Schwartz, Henrik Bengtsson, Peter Dalgaard +7 more

#
[on the R-SIG-Mac  mailing list] :	  

    > Hi all,
    > With R version 3.1.0 on OSX, using either the Snow Leopard or the Mavericks binary installation on a Mac with fully updated Mavericks, there has been a change in behavior since 3.0.3.

I've just written to R-core about this:

    R CMD Sweave

has *serious* problems also in the very simple case when it
should produce figures: They are not available after the
completion of R CMD Sweave,
confirming your  'deletes non tex files created upon batch mode exit'

So this is not related to OSX only, but also a big problem 
at least on other *nix descendent platforms, such as Linux.

In short, failure (no graphic produce in the example below) by

   R CMD Sweave foo.Rnw

but all is fine with using Sweave(), as you Marc, noted, and hence with  

   Rscript -e 'Sweave("foo.Rnw")'

... and to answer your question:  
No this was not intended and is probably one of the bigger / 
most embarrassing bugs in a newly released version of R
in my view: 

Basically  'R CMD Sweave' is partly broken in R 3.1.0.
Yes, this should never have happened.

I had to partially revert the R 3.1.0 installation here (our
statistics dept), by making R 3.0.3 the default 'R'
for now, as we are relying on 'R CMD Sweave ..' in many places.

Personally I've never noticed the problem, as I seem to always
Sweave from ESS (Emacs speaks statistics) which calls  Sweave()
in R, and that works fine, also in  R 3.1.0, as you've already
noted 

Martin Maechler, ETH Zurich

%------------------------------------------------------------
\documentclass[12pt]{article}
\usepackage{Sweave}
\begin{document}  Just a simple graphic 

<<qqnorm, fig=TRUE>>= 
qqnorm(rnorm(20))
@ 

and that's all, folks!
\end{document}
%------------------------------------------------------------

    > I have a master .Rnw file which runs a series of outputs from multiple R code files, each called in BATCH mode using system() from within the master .Rnw file. The output of the R code files go to separate text files in order to catch some of the function call output that would not otherwise be included in the resultant .tex file due to output redirection.

    > Those text files are then included in the resultant .tex file using, for example:

    > \lstinputlisting[caption={}]{test.out}

    > directives which are included in the .Rnw source file.

    > A simple example to replicate the observed behaviors.

    > The test.Rnw file content:

    > %% R CMD Sweave test.Rnw
    > <<results=tex,echo=false>>=
    > system("R CMD BATCH test.R test.out")
    > @ 


    > The test.R file content:

    > options(echo = FALSE)
    > options(useFancyQuotes = FALSE)
    > installed.packages()


    > On version 3.0.3, the file test.out is created, along with test.tex. test.out contains the output of installed.packages(). I did not include the aforementioned listing directive in test.Rnw here for simplicity.

    > On version 3.1.0, the file test.out is created, but when the R CMD Sweave command exits and returns to the CLI in the console, test.out is deleted, presumably as part of a post R batch session clean up process. The file test.tex is retained.

    > I uninstalled 3.1.0 and reinstalled 3.0.3 and observed the prior behavior. I then tried clean installs of both the Snow Leopard and Mavericks 3.1.0 binaries, with the new behavior observed in both cases.

    > In reading the NEWS file for 3.1.0, there are multiple references to Sweave, but there is nothing explicit about this new behavior. 

    > I should note that when the .Rnw file is run from within an R 3.1.0 interactive session using:

    > Sweave("test.Rnw")

    > the test.out file is created and not deleted upon the function exit, or when exiting the R session back to the console. 

    > Thus, this new behavior seems to be limited to running Sweave from the CLI using R CMD. It is not clear to me if this new behavior is by design or perhaps an unintended consequence of changes elsewhere in Sweave processing or in the handling of R in BATCH mode.

    > When watching the folder where the file output activity takes place, I note that a file .build.timestamp is created and then deleted, which leads me to believe that the new functions fileSnapshot() and changedFiles() are being used in the course of processing R in BATCH mode. If that is correct and there is a clean up step that is occurring upon BATCH mode exit, which deletes all files and folders other than .tex files, is it possible that this process is being overly aggressive? Or if not, is there a way to not have these files deleted or have a protected folder where these files can be retained?

    > Thanks for any insights.

    > Regards,

    > Marc Schwartz
#
Hi Martin,

Thanks for your confirmation on this.

I normally do not use R CMD Sweave, as I too run under ESS in normal day to day operations. This finding was a quirk of having a particular Rnw document that I occasionally run using R CMD Sweave and I had done so over the weekend, realizing this behavior.

Thanks again.

Regards,

Marc
On Apr 14, 2014, at 7:28 AM, Martin Maechler <maechler at stat.math.ethz.ch> wrote:

            
#
Until fixed, one way to fool R here without changing the vignette or
anything else seems to be (verified on Windows):

mkdir figures
set SWEAVE_OPTIONS=prefix.string=figures/fig
R CMD Sweave foo.Rnw

That obviously has some limitations.

/Henrik


On Mon, Apr 14, 2014 at 5:28 AM, Martin Maechler
<maechler at stat.math.ethz.ch> wrote:
#
On 14/04/2014, 10:23 AM, Marc Schwartz wrote:
This sounds like an argument for dropping R CMD Sweave, rather than 
fixing it.  The bug was introduced in July, 2013, and nobody noticed it 
because so few people use that feature, and apparently nobody who does 
use it bothers to test pre-release versions.

Duncan Murdoch
#
On 15 Apr 2014, at 13:14 , Duncan Murdoch <murdoch.duncan at gmail.com> wrote:

            
I'd say that that cat is out of the bag. There are probably umpteen documents around suggesting "R CMD Sweave". As people use Sweave only sporadically, it could take years before the old usage got stamped out. And anyways, the command format is the obvious way to generate documents in scripts and makefiles, isn't it?

-pd
#
On 15/04/2014 8:12 AM, peter dalgaard wrote:
I use Rscript, and I expect most others do too, but I was wrong about 
how long this went unreported.  Martin Morgan reported it in February in 
R-devel and I didn't notice.  He even gave a link to it in a message in 
March on another topic; I replied to the March message, but didn't 
follow the link.

We discourage people from using bugs.r-project.org for pre-release 
issues; perhaps we shouldn't do that.

Duncan Murdoch
#
Regardless of the venue, the cost for submitting an issue that doesn't
turn out to be a bug is quite high - you're likely to get a nasty
message. If you want more people to submit bug reports, I think you
have to be prepared to receive reports about things that users think
are bugs, but developers do not. If you want to encourage people to
submit bugs, then you have to make it a pleasant (or at least not
unpleasant) experience.

Hadley
#
I have, at times, been a heavy user of R CMD Sweave and would prefer to see it fixed!   

I encountered this error while preparing notes for a guest lecture last fall, and was too busy to identify the source of the error, so worked around it (thank goodness for version control, making file restores easy) rather than reporting the bug?.

-Greg
On Apr 15, 2014, at 10:11 AM, Hadley Wickham <h.wickham at gmail.com> wrote:

            
#
On Tue, 15 Apr 2014, peter dalgaard wrote:
A majority of my Sweave building is done by R CMD Sweave, I just 
infrequently build Sweave documents. I've started transitioning to RStudio 
for this but only in certain situations.
#

        
> On 15/04/2014 8:12 AM, peter dalgaard wrote:
>> On 15 Apr 2014, at 13:14 , Duncan Murdoch <murdoch.duncan at gmail.com> wrote:
>>
>> > On 14/04/2014, 10:23 AM, Marc Schwartz wrote:
>> >> Hi Martin,
    >> >>
    >> >> Thanks for your confirmation on this.
    >> >>
    >> >> I normally do not use R CMD Sweave, as I too run under ESS in normal day to day operations. This finding was a quirk of having a particular Rnw document that I occasionally run using R CMD Sweave and I had done so over the weekend, realizing this behavior.
    >> >
    >> > This sounds like an argument for dropping R CMD Sweave, rather than fixing it.  The bug was introduced in July, 2013, and nobody noticed it because so few people use that feature, and apparently nobody who does use it bothers to test pre-release versions.
    >> >
    >> 
    >> I'd say that that cat is out of the bag. There are probably umpteen documents around suggesting "R CMD Sweave". As people use Sweave only sporadically, it could take years before the old usage got stamped out. And anyways, the command format is the obvious way to generate documents in scripts and makefiles, isn't it?
    >> 

    > I use Rscript, and I expect most others do too, but I was wrong about 
    > how long this went unreported.  Martin Morgan reported it in February in 
    > R-devel and I didn't notice.  He even gave a link to it in a message in 
    > March on another topic; I replied to the March message, but didn't 
    > follow the link.

    > We discourage people from using bugs.r-project.org for pre-release 
    > issues; perhaps we shouldn't do that.

    > Duncan Murdoch

I agree:  Rather we should encourage such volunteer testers reporting
such problems.

The 'R version' is already part of the bug report form;
I'm not sure we can easily *list* all the "R version == R-devel"
bugs easily via web interface, but even w/o that,
we should rather be glad for such reports.

PS: I'm currently testing a patch where 'R CMD Sweave' will
    revert to not deleting anything after running the R code by default.

Martin Maechler
2 days later
#
[................]

    > PS: I'm currently testing a patch where 'R CMD Sweave' will
    > revert to not deleting anything after running the R code by default.

    > Martin Maechler

Some may have noted that R-devel, since svn revsion 65401 (= 2014-04-17 12:19:44 +0200)
now is patched, with log message
The daily (source!) snapshots of R devel now also contain it.

We plan to port the patch to 'R 3.1.0 patched' (to become 3.1.1
in the future) after the Easter holidays...
and would be glad if some volunteers could support development
of R by testing this (or newer) version of R-devel.

Martin Maechler, ETH Zurich
#
On Apr 19, 2014, at 8:05 AM, Martin Maechler <maechler at stat.math.ethz.ch> wrote:

            
Hi Martin,

Thanks for this.

I removed 3.1.0 release on my Mac and cleanly installed:

  R Under development (unstable) (2014-04-17 r65403) -- "Unsuffered Consequences"

from Simon's binary site. This is using the Mavericks binary, albeit on Simon's site, it indicates r65407.

I ran the prior .Rnw file that started this thread and it now works fine, as the created 'non-tex' files are retained upon completion of the R CMD Sweave run.

Thanks for the efforts in fixing this issue.

Regards,

Marc Schwartz
1 day later
#
> On Apr 19, 2014, at 8:05 AM, Martin Maechler
> <maechler at stat.math.ethz.ch> wrote:
>> 
    >>>>>>> Martin Maechler <maechler at stat.math.ethz.ch> on Thu,
    >>>>>>> 17 Apr 2014 11:22:04 +0200 writes:
    >> 
    >> [................]
    >> 
    >>> PS: I'm currently testing a patch where 'R CMD Sweave'
    >>> will revert to not deleting anything after running the R
    >>> code by default.
    >> 
    >>> Martin Maechler
    >> 
    >> Some may have noted that R-devel, since svn revsion 65401
    >> (= 2014-04-17 12:19:44 +0200) now is patched, with log
    >> message
    >> 
    >>> R CMD Sweave must not delete files by default;
    >>> buildVignette(*, keep); update (and fix/clarify)
    >>> documentation; cosmetic (& speedup in buildVignettes())
    >> 
    >> The daily (source!) snapshots of R devel now also contain
    >> it.
    >> 
    >> We plan to port the patch to 'R 3.1.0 patched' (to become
    >> 3.1.1 in the future) after the Easter holidays...  and
    >> would be glad if some volunteers could support
    >> development of R by testing this (or newer) version of
    >> R-devel.
    >> 
    >> Martin Maechler, ETH Zurich


    > Hi Martin,

    > Thanks for this.

    > I removed 3.1.0 release on my Mac and cleanly installed:

    >   R Under development (unstable) (2014-04-17 r65403) --
    > "Unsuffered Consequences"

    > from Simon's binary site. This is using the Mavericks
    > binary, albeit on Simon's site, it indicates r65407.

    > I ran the prior .Rnw file that started this thread and it
    > now works fine, as the created 'non-tex' files are
    > retained upon completion of the R CMD Sweave run.

Thank you, Marc, for the confirmation.

    > Thanks for the efforts in fixing this issue.

you are welcome, and thanked again together with everyone who
spends the little extra time for checking / using "the next
version of R" -- in general, i.e., on a regular basis.

Martin

    > Regards,
    > Marc Schwartz
#
On Sun, Apr 20, 2014 at 11:19 AM, Martin Maechler
<maechler at stat.math.ethz.ch> wrote:
With the risk of starting another flamewar I would like to point out
that a natural way to encourage use of r-patched/r-dev would be
extending the R release cycle to CRAN packages as was proposed on this
mailing list a couple of weeks ago.

If install.packages would default to downloading frozen/stable CRAN
packages in r-release, and download the very latest CRAN packages in
r-patched/r-dev, then most developers would probably switch to r-dev.
Thereby they would automatically be using the latest ("devel")
versions of base packages in the same way as they currently test their
script/project/package to work with the latest ("current") versions of
CRAN packages.
#
On 20/04/2014, 6:43 PM, Jeroen Ooms wrote:
I doubt if that would be true.  I'd guess most everybody would use 
r-patched.  This would have the effect that nobody would have 
reproducible research (since r-patched builds aren't archived), and few 
would test r-devel.

Duncan Murdoch