tools:: extracting pkg dependencies from DCF

Gabriel,

I am trying to design generic solution that could be applied to
arbitrary package. Therefore I went with the latter solution you
proposed.
If we wouldn't have to exclude base packages, then its a 3 liner

file.copy("DESCRIPTION", file.path(tdir<-tempdir(), "PACKAGES"));
db<-available.packages(paste0("file://", tdir));
utils::install.packages(tools::package_dependencies("pkgname", db,
which="most")[[1L]])

As you noticed, we still have to filter out base packages. Otherwise
it won't be a robust utility that can be used in CI. Therefore we have
to add a call to tools:::.get_standard_package_names() which is an
internal function (as of now). Not only complicating the call but also
putting the functionality outside of safe use.

Considering above, don't you agree that the following one liner could
nicely address the problem? The problem that hundreds/thousands of
packages are now addressing in their CI scripts by using a third party
packages.

utils::install.packages(packages.dcf("DESCRIPTION", which="most"))

It is hard to me to understand why R members don't consider this basic
functionality to be part of base R. Possibly they just don't need it
themselves. Yet isn't this sufficient that hundreds/thousands of
packages does need this functionality?

Best regards,
Jan

On Mon, Oct 17, 2022 at 8:39 AM Jan Gorecki <j.gorecki at wit.edu.pl> wrote:

Gabriel and Simon

I completely agree with what you are saying.
The thing is that obtaining recursive deps, all/most whatever, is

already well supported in core R. What is missing is just this single
functionality I am requesting.

If you will look into the branch you can see there is mirror.packages

function meant to mirror a slice of CRAN. It is doing exactly what you
described: package_dependencies; to obtain recursive deps, then download
all, etc.

I would love to have this function provided by core R as well, but we

need to start somewhere.

There are other use cases as well.
For example CI, where one wants to install all/most dependencies and

then run R CMD check. Then we don't worry about recursive deps are they
will be resolved automatically.

I don't think it's reasonable to force users to use 3rd party packages

to handle such a common and simple use case. Otherwise one has to hard code
deps in CI script. Not robust at all.

packages.dcf and repos.dcf makes all that way easier, and are solid base

for building customized orchestration like mirroring slice of CRAN.

Best regards
Jan

On Sun, Oct 16, 2022, 01:31 Simon Urbanek <simon.urbanek at r-project.org>

wrote:

Jan,

I think using a single DCF as input is not very practical and would not

be useful in the context you describe (creating self contained repos) since
they typically concern a list of packages, but essentially splitting out
the part of install.packages() which determines which files will be pulled
from where would be very useful as it would be trivial to use it to create
repository (what we always do in corporate environments) instead of
installing the packages. I suspect that install packages is already too
complex so instead of adding a flag to install.packages one could move that
functionality into a separate function - we all do that constantly for the
sites we manage, so it would be certainly something worthwhile.

Cheers,
Simon

On Oct 15, 2022, at 7:14 PM, Jan Gorecki <j.gorecki at wit.edu.pl>

wrote:

Hi Gabriel,

It's very nice usage you provided here. Maybe instead of adding new
function we could extend packages_depenedncies then? To accept file

path to

dsc file.

What about repos.dcf? Maybe additional repositories could be an

attribute

attached to returned character vector.

The use case is to, for a given package sources, obtain its

dependencies,

so one can use that for installing them/mirroring CRAN subset, or

whatever.

The later is especially important for a production environment where

one

wants to have fixed version of packages, and mirroring relevant

subset of

CRAN is the most simple, and IMO reliable, way to manage such

environment.

Regards
Jan

On Fri, Oct 14, 2022, 23:34 Gabriel Becker <gabembecker at gmail.com>

wrote:

Hi Jan and Jan,

Can you explain a little more what exactly you want the

non-recursive,

non-version aware dependencies from an individual package for?

Either way package_dependencies will do this for you* with a little
"aggressive convincing". It wants output from available.packages,

but who

really cares what it wants? It's a function and we are people :)

library(tools)
db <- read.dcf("~/gabe/checkedout/rtables_clean/DESCRIPTION")
package_dependencies("rtables", db, which = intersect(c("Depends",

"Suggests", "Imports", "LinkingTo"), colnames(db)))
$rtables
[1] "methods"    "magrittr"   "formatters" "dplyr"      "tibble"
[6] "tidyr"      "testthat"   "xml2"       "knitr"      "rmarkdown"
[11] "flextable"  "officer"    "stats"      "htmltools"  "grid"


The only gotcha that I see immediately is that "LinkingTo" isn't

always

there (whereas it is with real output from available.packages). If

you

know your package doesn't have that (or that it does) at call time ,

this

becomes a one-liner:

package_dependencies("rtables", db =
read.dcf("~/gabe/checkedout/rtables_clean/DESCRIPTION"), which =
c("Depends", "Suggests", "Imports"))
$rtables
[1] "methods"    "magrittr"   "formatters" "dplyr"      "tibble"
[6] "tidyr"      "testthat"   "xml2"       "knitr"      "rmarkdown"
[11] "flextable"  "officer"    "stats"      "htmltools"  "grid"

You can also trick it a slightly different way by giving it what it
actually wants

tdir <- tempdir()
file.copy("~/gabe/checkedout/rtables_clean/DESCRIPTION",

file.path(tdir,

"PACKAGES"))
[1] TRUE

avl <- available.packages(paste0("file://", tdir))
library(tools)
package_dependencies("rtables", avl)

$rtables
[1] "methods"    "magrittr"   "formatters" "stats"      "htmltools"
[6] "grid"

package_dependencies("rtables", avl, which = "all")

$rtables
[1] "methods"    "magrittr"   "formatters" "stats"      "htmltools"
[6] "grid"       "dplyr"      "tibble"     "tidyr"      "testthat"
[11] "xml2"       "knitr"      "rmarkdown"  "flextable"  "officer"

So the only real benefits I see that we'd be picking up here is

automatic

filtering by priority, and automatic extraction of the package name

from

the DESCRIPTION file. I'm not sure either of those warrant a new

exported

function that R-core has to maintain forever.

Best,
~G

* I haven't tested this across all OSes, but I dont' know of any

reason it

wouldn't work generally.

On Fri, Oct 14, 2022 at 2:33 PM Jan Gorecki <j.gorecki at wit.edu.pl>

wrote:

Hello Jan,

Thanks for confirming about many packages reinventing this missing
functionality.
packages.dcf was not meant handle versions. It just extracts names

of

dependencies... Yes, such a simple thing, yet missing in base R.

Versions of packages can be controlled when setting up R pkgs repo.

This

is
how I used to handle it. Making a CRAN subset mirror of fixed

version

pkgs.
BTW. function for that is also included in mentioned branch. I am

just not

proposing it, to increase the chance of having at least this simple,
missing, functionality merged.

Best
Jan

On Fri, Oct 14, 2022, 15:14 Jan Net?k <netikja at gmail.com> wrote:

Hello Jan,

I have seen many packages that implemented dependencies

"extraction" on

their own for internal purposes and today I was doing exactly that

for

mine. It's not a big deal using read.dcf on DESCRIPTION. It was

sufficient

for me, but I had to take care of some \n chars (the overall

returned

value

has some rough edges, in my opinion). However, the function from

the

branch

seems to not care about version requirements, which are crucial

for me.

Maybe that is something to reconsider before merging.

Best,
Jan

p? 14. 10. 2022 v 2:27 odes?latel Jan Gorecki <

j.gorecki at wit.edu.pl>

napsal:

Dear R devs,

I would like to raise a request for a simple helper function.
Utility function to extract package dependencies from DESCRIPTION

file.

I do think that tools package is better place, for such a

fundamental

functionality, than community packages.

tools pkg seems perfect fit (having already great function
write_PACKAGES).

Functionality I am asking for is already in R svn repository since

2016,

in
a branch tools4pkgs. Function is called 'packages.dcf'.
Another one 'repos.dcf' would be a good functional complementary

to it.

Those two simple helper functions really makes it easier for

organizations

to glue together usage of their own R packages repos and CRAN

repo in a

smooth way. That could possibly help to offload CRAN from new

submissions.

gh mirror link for easy preview:

https://github.com/wch/r-source/blob/tools4pkgs/src/library/tools/R/packages.R#L419

Regards
Jan Gorecki

       [[alternative HTML version deleted]]

       [[alternative HTML version deleted]]

      [[alternative HTML version deleted]]