Should package version requirements assume installation from sources?

On 13/09/2022 5:45 p.m., Mikael Jagan wrote:

[Arguably also appropriate for R-package-devel, but posted to R-devel
       as the discussion is aimed primarily at "experts" ... ]

We, the authors of Matrix, have encountered a somewhat subtle issue
induced by caching of S4 classes and methods in package namespaces.

The namespaces of three reverse dependent packages (SeuratObject, conText,
mcmcsae) cache the formal definition of our virtual class Matrix (and some
subclasses).  For example:

   > ns <- asNamespace("SeuratObject")
   > grep("^[.]__C__.*Matrix$", names(ns), value = TRUE)

[1] ".__C__dMatrix"       ".__C__compMatrix"    ".__C__AnyMatrix"
[4] ".__C__generalMatrix" ".__C__CsparseMatrix" ".__C__sparseMatrix"
[7] ".__C__dsparseMatrix" ".__C__Matrix"

The cached definition (which includes a _validity method_) is obtained from
the version of Matrix available when the reverse dependent package was built
from sources.  For example, if SeuratObject was built under Matrix 1.4-1,
then we get:

   > getValidity(ns$.__C__Matrix)

function (object)
{
       if (!isTRUE(r <- .Call(Dim_validate, object, "Matrix")))
           r
       else .Call(dimNames_validate, object)
}
<bytecode: 0x11e7ca508>
<environment: namespace:Matrix>

whereas if SeuratObject was built under Matrix >= 1.5-0, then we get:

   > getValidity(ns$.__C__Matrix)

function (object)
.Call(Matrix_validate, object)
<bytecode: 0x107dc1698>
<environment: namespace:Matrix>

There are two "questions" here:

1.  The symbol 'Matrix_validate' is not defined until Matrix 1.5-0.
       Is it necessary, for this reason alone, for SeuratObject to have
       'Imports: Matrix (>= 1.5-0)'?  Or can SeuratObject continue using
       'Imports: Matrix (>= 1.3-3)', at the risk of errors like

       > Error: object 'Matrix_validate' not found

       (as already seen here: https://stackoverflow.com/questions/73700130)?

       Note that this error would not occur for anyone installing SeuratObject
       from sources, unless they decide to _downgrade_ Matrix after doing so.
       Hence this primarily concerns Windows and macOS where R users would
       typically install a binary built by CRAN (i.e., not on their system).

       We are aware that package TMB tests in .onLoad() that the current Matrix
       version is equal to or greater than the version available at build time,
       thus avoiding a "strict" version requirement, but do not want this practice
       to spread ...

2.  For how long should Matrix retain the superceded 'Dim_validate' and
       'dimNames_validate', in order to ensure that "stale" cached validity
       methods continue to work?

We hope that this discussion will highlight the potential ramifications
of importing classes and methods from other packages, and having one's
classes and methods imported _by_ other packages, especially for version
requirements.

This sounds like a bug or bad design in the S4 system, i.e. caching
things without a way to detect or update when the cache becomes stale.

Is it really necessary to cache things as part of the binary package, or
could they be put in place when needed using lazy loading, getting a
copy from the loaded copy of Matrix?

Duncan Murdoch