A demonstrated shortcoming of the R package management system

CRAN, by relying on the powerful package management system that is part of R,
provides an unparalleled framework for extending R with nearly 20k packages.

We recently encountered an issue that highlights a missing element in the
otherwise outstanding package management system. So we would like to start a
discussion about enhancing its feature set. As shown below, a mechanism to
force reinstallation of packages may be needed.

A demo is included below, it is reproducible in a container. We find the
easiest/fastest reproduction is by saving the code snippet below in the
current directory as eg 'matrixIssue.R' and have it run in a container as

    docker run --rm -ti -v `pwd`:/mnt rocker/r2u Rscript /mnt/matrixIssue.R
   
This runs in under two minutes, first installing the older Matrix, next
installs SeuratObject, and then by removing the older Matrix making the
(already installed) current Matrix version the default. This simulates a
package update for Matrix. Which, as the final snippet demonstrates, silently
breaks SeuratObject as the cached S4 method Csparse_validate is now missing.
So when SeuratObject was installed under Matrix 1.5.1, it becomes unuseable
under Matrix 1.6.0.

What this shows is that a call to update.packages() will silently corrupt an
existing installation.  We understand that this was known and addressed at
CRAN by rebuilding all binary packages (for macOS and Windows).

But it leaves both users relying on source installation as well as
distributors of source packages in a dire situation. It hurt me three times:
my default R installation was affected with unit tests (involving
SeuratObject) silently failing. It similarly broke our CI setup at work.  And
it created a fairly bad headache for the Debian packaging I am involved with
(and I surmise it affects other distro similarly).

It would be good to have a mechanism where a package, when being upgraded,
could flag that 'more actions are required' by the system (administrator).
We think this example demonstrates that we need such a mechanism to avoid
(silently !!) breaking existing installations, possibly by forcing
reinstallation of other packages.  R knows the package dependency graph and
could trigger this, possibly after an 'opt-in' variable the user / admin
sets.

One possibility may be to add a new (versioned) field 'Breaks:'. Matrix could
then have added 'Breaks: SeuratObject (<= 4.1.3)' preventing an installation
of Matrix 1.6.0 when SeuratObject 4.1.3 (or earlier) is present, but
permitting an update to Matrix 1.6.0 alongside a new version, say, 4.1.4 of
SeuratObject which could itself have a versioned Depends: Matrix (>= 1.6.0).

Regards,  Dirk


## Code example follows. Recommended to run the rocker/r2u container.
## Could also run 'apt update -qq; apt upgrade -y' but not required
## Thanks to my colleague Paul Hoffman for the core of this example

## now have Matrix 1.6.0 because r2u and CRAN remain current but we can install an older Matrix
remotes::install_version('Matrix', '1.5.1')

## we can confirm that we have Matrix 1.5.1
packageVersion("Matrix")

## we now install SeuratObject from source and to speed things up we first install the binary
install.packages("SeuratObject")   # in this container via bspm/r2u as binary
## and then force a source installation (turning bspm off) _while Matrix is at 1.5.1_
if (requireNamespace("bspm", quietly=TRUE) bspm::disable()
Sys.setenv(PKG_CXXFLAGS='-Wno-ignored-attributes') 	# Eigen compilation noise silencer
install.packages('SeuratObject')

## we now remove the Matrix package version 1.5.1 we installed into /usr/local leaving 1.6.0
remove.packages("Matrix")
packageVersion("Matrix")

## and we now run a bit of SeuratObject code that is now broken as Csparse_validate is gone
suppressMessages(library(SeuratObject))
data('pbmc_small')
graph <- pbmc_small[['RNA_snn']]
class(graph)
getClass('Graph')
show(graph) # this fails