Skip to content

[R-pkg-devel] Question regarding listing base and recommended packages programmatically and efficiently

8 messages · Tony Wilkes, Thierry Onkelinx, Duncan Murdoch +4 more

Tony Wilkes
# 
Dear all,

In my R package that I'm developing, I use `installed.packages(priority = "base")` to programmatically get all core/base R packages (i.e. base, stats, etc.). And similarly, I use installed.packages(priority = "recommended")?` to programmatically get the recommended R packages (i.e. mgcv, lattice, etc.).

However, CRAN has requested to not use `installed.packages()`, as it is slow. I fully get and agree with that assesment. I used installed.packages()?` anyway because I could not find a better, fool-proof alternative.

Nonetheless, I was asked to change this code for optimalisation. So I would like to ask: how do I programmatically get all base/core R packages safely and efficiently, but without using `installed.packages()`? And the same question for recommended R packages. I have of course Googled it, and looked at R's documentation (though R's documentation is large, so it's easy to miss something); no solution found. So if any of you has a smart idea: I'm all ears.

Thank you in advance.

Kind regards,

Tony.
Thierry Onkelinx
# 
Dear Tony,

Much will depend on what information you need from the output of
installed.packages()?

Best regards,

ir. Thierry Onkelinx
Statisticus / Statistician

Vlaamse Overheid / Government of Flanders
INSTITUUT VOOR NATUUR- EN BOSONDERZOEK / RESEARCH INSTITUTE FOR NATURE AND
FOREST
Team Biometrie & Kwaliteitszorg / Team Biometrics & Quality Assurance
thierry.onkelinx at inbo.be
Havenlaan 88 bus 73, 1000 Brussel
www.inbo.be

///////////////////////////////////////////////////////////////////////////////////////////
To call in the statistician after the experiment is done may be no more
than asking him to perform a post-mortem examination: he may be able to say
what the experiment died of. ~ Sir Ronald Aylmer Fisher
The plural of anecdote is not data. ~ Roger Brinner
The combination of some data and an aching desire for an answer does not
ensure that a reasonable answer can be extracted from a given body of data.
~ John Tukey
///////////////////////////////////////////////////////////////////////////////////////////

<https://www.inbo.be>


Op do 12 okt 2023 om 14:34 schreef Tony Wilkes <tony_a_wilkes at outlook.com>:

  
  


  


      

    

    

      
      

        


  

        

      Duncan Murdoch
    

    

      
      #
    

  


  

      
It would be much faster (but slightly less reliable) to use 
list.files(.libPaths()) to get the names of all installed packages, and 
then filter them to the known list of base and recommended packages, 
which changes very rarely.

Duncan Murdoch

        
On 12/10/2023 8:34 a.m., Tony Wilkes wrote:

            

      
      
      
    

        

      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Mikael Jagan
    

    

      
      #
    

  


  

      
Maybe something like this:

     > isRecommendedPkg <- utils:::isBasePkg
     > body(isRecommendedPkg)[[c(3L, 3L)]] <- "recommended"
     > installed <- unique(list.files(.libPaths()))
     > installed[vapply(installed, isRecommendedPkg, NA)]
      [1] "KernSmooth" "MASS"       "Matrix"     "boot"       "class"
      [6] "cluster"    "codetools"  "foreign"    "lattice"    "mgcv"
     [11] "nlme"       "nnet"       "rpart"      "spatial"    "survival"

where in your package you would define isRecommendedPkg "manually".

Another (but quite undocumented and so maybe not "recommended" :-))
possibility is this:

     > mk <- file.path(R.home("share"), "make", "vars.mk")
     > pp <- sub("^.*= +", "", grep("^R_PKGS_RECOMMENDED", readLines(mk), value 
= TRUE))
     > sort(strsplit(pp, " ")[[1L]])
      [1] "KernSmooth" "MASS"       "Matrix"     "boot"       "class"
      [6] "cluster"    "codetools"  "foreign"    "lattice"    "mgcv"
     [11] "nlme"       "nnet"       "rpart"      "spatial"    "survival"

I grepped around and did not find variables in any base namespace containing
the names of these packages.  It wouldn't be too hard to define such variables
when R is configured/built, but maybe there are "reasons" to not do that ... ?

Mikael

 > It would be much faster (but slightly less reliable) to use
 > list.files(.libPaths()) to get the names of all installed packages, and
 > then filter them to the known list of base and recommended packages,
 > which changes very rarely.
 >
 > Duncan Murdoch
 >

        
> On 12/10/2023 8:34 a.m., Tony Wilkes wrote:

        
> > Dear all,
 > >
 > > In my R package that I'm developing, I use `installed.packages(priority = 
"base")` to programmatically get all core/base R packages (i.e. base, stats, 
etc.). And similarly, I use installed.packages(priority = "recommended")?` to 
programmatically get the recommended R packages (i.e. mgcv, lattice, etc.).
 > >
 > > However, CRAN has requested to not use `installed.packages()`, as it is 
slow. I fully get and agree with that assesment. I used installed.packages()?` 
anyway because I could not find a better, fool-proof alternative.
 > >
 > > Nonetheless, I was asked to change this code for optimalisation. So I would 
like to ask: how do I programmatically get all base/core R packages safely and 
efficiently, but without using `installed.packages()`? And the same question for 
recommended R packages. I have of course Googled it, and looked at R's 
documentation (though R's documentation is large, so it's easy to miss 
something); no solution found. So if any of you has a smart idea: I'm all ears.
 > >
 > > Thank you in advance.
 > >
 > > Kind regards,
 > >
 > > Tony.

  
  


  


      

    

    

      
      

        


  

        

      Ivan Krylov
    

    

      
      #
    

  


  

      
On Thu, 12 Oct 2023 11:32:24 -0400

        
Mikael Jagan <jaganmn2 at gmail.com> wrote:

        

            

      
      
      
    

        
tools:::.get_standard_package_names does that at package installation
time, but it's still not public API.

A call to installed.packages() may take a long while because it has to
list files in every library (some of which can be large and/or
network-mounted) and parse each Meta/package.rds file, but at least
list.files() is faster than that.

If I had to make a choice at this point, I would hard-code the list of
packages, but a better option may surface once we know what Tony needs
the package lists for.

  
    

      
      
    

  


  


      

    

    
1 day later

    

      
      

        


  

        

      Martin Maechler
    

    

      
      #
    

  


  

          

      
      
      
    

        
> On Thu, 12 Oct 2023 11:32:24 -0400

        
> Mikael Jagan <jaganmn2 at gmail.com> wrote:

        
>> > mk <- file.path(R.home("share"), "make", "vars.mk")
    >> > pp <- sub("^.*= +", "", grep("^R_PKGS_RECOMMENDED",
    >> > readLines(mk), value = TRUE))
    >> > sort(strsplit(pp, " ")[[1L]])  
    >> [1] "KernSmooth" "MASS"       "Matrix"     "boot"       "class"
    >> [6] "cluster"    "codetools"  "foreign"    "lattice"    "mgcv"
    >> [11] "nlme"       "nnet"       "rpart"      "spatial" "survival"
    >> 
    >> I grepped around and did not find variables in any base namespace
    >> containing the names of these packages.  It wouldn't be too hard to
    >> define such variables when R is configured/built, but maybe there are
    >> "reasons" to not do that ... ?

    > tools:::.get_standard_package_names does that at package installation
    > time, but it's still not public API.

Within R-core, we have somewhat discussed this, and a few
minutes ago I committed a "public API" version of the above,
called
   standard_package_names()

to R-devel (svn rev 85329), and hence probably in next year's
April release of R.


    > A call to installed.packages() may take a long while because it has to
    > list files in every library (some of which can be large and/or
    > network-mounted) and parse each Meta/package.rds file, but at least
    > list.files() is faster than that.

The above is another issue that we've wanted to improve, as some
of you are aware,  notably thinking about caching the result
.. there has been work on this during the R Sprint @ Warwick a
couple of weeks ago,

 ==> https://github.com/r-devel/r-project-sprint-2023/issues/78

involving smart people and promising proposals (my personal view).

  > If I had to make a choice at this point, I would hard-code the list of
  > packages, but a better option may surface once we know what Tony needs
  > the package lists for.

  > -- 
  > Best regards,
  > Ivan


With thanks to the discussants here on R-devel,
and best regards,
Martin

--
Martin Maechler
ETH Zurich  and  R Core team

  
  


  


      

    

    

      
      

        


  

        

      Henrik Bengtsson
    

    

      
      #
    

  


  

      
On Sat, Oct 14, 2023 at 5:25?AM Martin Maechler

        
<maechler at stat.math.ethz.ch> wrote:

            

      
      
      
    

        
Excellent. Will it be supported on all OSes?  Because, there's
currently a source code comment saying the current implementation
might not work on MS Windows:

## we cannot assume that file.path(R.home("share"), "make", "vars.mk")
## is installed, as it is not on Windows
standard_package_names <-
.get_standard_package_names <-
local({
    lines <- readLines(file.path(R.home("share"), "make", "vars.mk"))
    lines <- grep("^R_PKGS_[[:upper:]]+ *=", lines, value = TRUE)
    out <- strsplit(sub("^R_PKGS_[[:upper:]]+ *= *", "", lines), " +")
    names(out) <-
        tolower(sub("^R_PKGS_([[:upper:]]+) *=.*", "\\1", lines))
    eval(substitute(function() {out}, list(out=out)), envir = topenv())
    })

/Henrik

            

      
      
      
    

        

      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Martin Maechler
    

    

      
      #
    

  


  

          

      
      
      
    

        
> On Sat, Oct 14, 2023 at 5:25?AM Martin Maechler

        
> <maechler at stat.math.ethz.ch> wrote:

        
>> 
    >> >>>>> Ivan Krylov
    >> >>>>>     on Thu, 12 Oct 2023 18:50:30 +0300 writes:
    >> 
    >> > On Thu, 12 Oct 2023 11:32:24 -0400

        
>> > Mikael Jagan <jaganmn2 at gmail.com> wrote:

        
>> 
    >> >> > mk <- file.path(R.home("share"), "make", "vars.mk")
    >> >> > pp <- sub("^.*= +", "", grep("^R_PKGS_RECOMMENDED",
    >> >> > readLines(mk), value = TRUE))
    >> >> > sort(strsplit(pp, " ")[[1L]])
    >> >> [1] "KernSmooth" "MASS"       "Matrix"     "boot"       "class"
    >> >> [6] "cluster"    "codetools"  "foreign"    "lattice"    "mgcv"
    >> >> [11] "nlme"       "nnet"       "rpart"      "spatial" "survival"
    >> >>
    >> >> I grepped around and did not find variables in any base namespace
    >> >> containing the names of these packages.  It wouldn't be too hard to
    >> >> define such variables when R is configured/built, but maybe there are
    >> >> "reasons" to not do that ... ?
    >> 
    >> > tools:::.get_standard_package_names does that at package installation
    >> > time, but it's still not public API.
    >> 
    >> Within R-core, we have somewhat discussed this, and a few
    >> minutes ago I committed a "public API" version of the above,
    >> called
    >> standard_package_names()
    >> 
    >> to R-devel (svn rev 85329), and hence probably in next year's
    >> April release of R.

    > Excellent. Will it be supported on all OSes?  Because, there's
    > currently a source code comment saying the current implementation
    > might not work on MS Windows:

    > ## we cannot assume that file.path(R.home("share"), "make", "vars.mk")
    > ## is installed, as it is not on Windows

Well, it does work nowadays on Windows {just checked}.
So the comment should probably be removed.

{{ I'm definitely not the expert, but AFAIK, the Windows
   installation of R has become considerably closer to the other platform
   ones, recently }}


    > standard_package_names <-
    > .get_standard_package_names <-
    > local({
    > lines <- readLines(file.path(R.home("share"), "make", "vars.mk"))
    > lines <- grep("^R_PKGS_[[:upper:]]+ *=", lines, value = TRUE)
    > out <- strsplit(sub("^R_PKGS_[[:upper:]]+ *= *", "", lines), " +")
    > names(out) <-
    > tolower(sub("^R_PKGS_([[:upper:]]+) *=.*", "\\1", lines))
    > eval(substitute(function() {out}, list(out=out)), envir = topenv())
    > })

    > /Henrik

    >> 
    >> 
    >> > A call to installed.packages() may take a long while because it has to
    >> > list files in every library (some of which can be large and/or
    >> > network-mounted) and parse each Meta/package.rds file, but at least
    >> > list.files() is faster than that.
    >> 
    >> The above is another issue that we've wanted to improve, as some
    >> of you are aware,  notably thinking about caching the result
    >> .. there has been work on this during the R Sprint @ Warwick a
    >> couple of weeks ago,
    >> 
    >> ==> https://github.com/r-devel/r-project-sprint-2023/issues/78
    >> 
    >> involving smart people and promising proposals (my personal view).
    >> 
    >> > If I had to make a choice at this point, I would hard-code the list of
    >> > packages, but a better option may surface once we know what Tony needs
    >> > the package lists for.
    >> 
    >> > --
    >> > Best regards,
    >> > Ivan
    >> 
    >> 
    >> With thanks to the discussants here on R-devel,
    >> and best regards,
    >> Martin
    >> 
    >> --
    >> Martin Maechler
    >> ETH Zurich  and  R Core team
    >> 
    >> ______________________________________________
    >> R-package-devel at r-project.org mailing list
    >> https://stat.ethz.ch/mailman/listinfo/r-package-devel