[RFC] A case for freezing CRAN
On Mar 19, 2014, at 22:45, Jeroen Ooms <jeroen.ooms at stat.ucla.edu> wrote:
On Wed, Mar 19, 2014 at 6:55 PM, Michael Weylandt <michael.weylandt at gmail.com> wrote:
Reading this thread again, is it a fair summary of your position to say "reproducibility by default is more important than giving users access to the newest bug fixes and features by default?" It's certainly arguable, but I'm not sure I'm convinced: I'd imagine that the ratio of new work being done vs reproductions is rather high and the current setup optimizes for that already.
I think that separating development from released branches can give us both reliability/reproducibility (stable branch) as well as new features (unstable branch). The user gets to pick (and you can pick both!). The same is true for r-base: when using a 'released' version you get 'stable' base packages that are up to 12 months old. If you want to have the latest stuff you download a nightly build of r-devel. For regular users and reproducible research it is recommended to use the stable branch. However if you are a developer (e.g. package author) you might want to develop/test/check your work with the latest r-devel.
I think where you are getting push back (e.g., Frank Harrell and Josh Ulrich) is from saying that 'stable' is the right branch for 'regular users.' And I tend to agree: I think most folks need features and bug fixes more than they need to reproduce a particular paper with no effort on their end.
I think that extending the R release cycle to CRAN would result both in more stable released versions of R, as well as more freedom for package authors to implement rigorous change in the unstable branch.
Not sure what exactly you mean by this sentence.
When writing a script that is part of a production pipeline, or sweave paper that should be reproducible 10 years from now, or a book on using R, you use stable version of R, which is guaranteed to behave the same over time.
Only if you never upgrade anything... But that's the case already, isn't it?
However when developing packages that should be compatible with the upcoming release of R, you use r-devel which has the latest versions of other CRAN and base packages.
What I'm trying to figure out is why the standard "install the following list of package versions" isn't good enough in your eyes?
Almost nobody does this because it is cumbersome and impractical. We can do so much better than this. Note that in order to install old packages you also need to investigate which versions of dependencies of those packages were used. On win/osx, users need to manually build those packages which can be a pain. All in all it makes reproducible research difficult and expensive and error prone. At the end of the day most published results obtain with R just won't be reproducible.
So you want CRAN to host old binaries ad infinitum? I think that's entirely reasonable/doable if (big if) storage and network are free.
Also I believe that keeping it simple is essential for solutions to be practical. If every script has to be run inside an environment with custom libraries, it takes away much of its power. Running a bash or python script in Linux is so easy and reliable that entire distributions are based on it. I don't understand why we make our lives so difficult in R.
Because for Debian style (stop the world on release) distro, there are no upgrades within a release. And that's only halfway reasonable because of Debian's shockingly good QA. It's certainly not true for, e.g., Arch. I've been looking at python incompatibilities across different RHEL versions lately. There's simply no way to get around explicit version pinning (either by release number or date, but when you have many moving pieces, picking a set of release numbers is much easier than finding a single day when they all happened to work together) if it has to work exactly as it used to.
In my estimation, a system where stable versions of R pull packages from a stable branch of CRAN will naturally resolve the majority of the reproducibility and reliability problems with R.
And what everyone else is saying is "if you want to reproduce results made with old software, download and use the old software." Both can me made to work -- it's just a matter of pros and cons of different defaults.
And in contrast to what some people here are suggesting it does not introduce any limitations. If you want to get the latest stuff, you either grab a copy of r-devel, or just enable the testing branch and off you go. Debian 'testing' works in a similar way, see http://www.debian.org/devel/testing.