Dear R devs, I would like to raise a request for a simple helper function. Utility function to extract package dependencies from DESCRIPTION file. I do think that tools package is better place, for such a fundamental functionality, than community packages. tools pkg seems perfect fit (having already great function write_PACKAGES). Functionality I am asking for is already in R svn repository since 2016, in a branch tools4pkgs. Function is called 'packages.dcf'. Another one 'repos.dcf' would be a good functional complementary to it. Those two simple helper functions really makes it easier for organizations to glue together usage of their own R packages repos and CRAN repo in a smooth way. That could possibly help to offload CRAN from new submissions. gh mirror link for easy preview: https://github.com/wch/r-source/blob/tools4pkgs/src/library/tools/R/packages.R#L419 Regards Jan Gorecki
tools:: extracting pkg dependencies from DCF
13 messages · Jan Netík, Gabriel Becker, Simon Urbanek +1 more
Hello Jan, I have seen many packages that implemented dependencies "extraction" on their own for internal purposes and today I was doing exactly that for mine. It's not a big deal using read.dcf on DESCRIPTION. It was sufficient for me, but I had to take care of some \n chars (the overall returned value has some rough edges, in my opinion). However, the function from the branch seems to not care about version requirements, which are crucial for me. Maybe that is something to reconsider before merging. Best, Jan p? 14. 10. 2022 v 2:27 odes?latel Jan Gorecki <j.gorecki at wit.edu.pl> napsal:
Dear R devs, I would like to raise a request for a simple helper function. Utility function to extract package dependencies from DESCRIPTION file. I do think that tools package is better place, for such a fundamental functionality, than community packages. tools pkg seems perfect fit (having already great function write_PACKAGES). Functionality I am asking for is already in R svn repository since 2016, in a branch tools4pkgs. Function is called 'packages.dcf'. Another one 'repos.dcf' would be a good functional complementary to it. Those two simple helper functions really makes it easier for organizations to glue together usage of their own R packages repos and CRAN repo in a smooth way. That could possibly help to offload CRAN from new submissions. gh mirror link for easy preview: https://github.com/wch/r-source/blob/tools4pkgs/src/library/tools/R/packages.R#L419 Regards Jan Gorecki [[alternative HTML version deleted]]
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Hello Jan, Thanks for confirming about many packages reinventing this missing functionality. packages.dcf was not meant handle versions. It just extracts names of dependencies... Yes, such a simple thing, yet missing in base R. Versions of packages can be controlled when setting up R pkgs repo. This is how I used to handle it. Making a CRAN subset mirror of fixed version pkgs. BTW. function for that is also included in mentioned branch. I am just not proposing it, to increase the chance of having at least this simple, missing, functionality merged. Best Jan
On Fri, Oct 14, 2022, 15:14 Jan Net?k <netikja at gmail.com> wrote:
Hello Jan, I have seen many packages that implemented dependencies "extraction" on their own for internal purposes and today I was doing exactly that for mine. It's not a big deal using read.dcf on DESCRIPTION. It was sufficient for me, but I had to take care of some \n chars (the overall returned value has some rough edges, in my opinion). However, the function from the branch seems to not care about version requirements, which are crucial for me. Maybe that is something to reconsider before merging. Best, Jan p? 14. 10. 2022 v 2:27 odes?latel Jan Gorecki <j.gorecki at wit.edu.pl> napsal:
Dear R devs, I would like to raise a request for a simple helper function. Utility function to extract package dependencies from DESCRIPTION file. I do think that tools package is better place, for such a fundamental functionality, than community packages. tools pkg seems perfect fit (having already great function write_PACKAGES). Functionality I am asking for is already in R svn repository since 2016, in a branch tools4pkgs. Function is called 'packages.dcf'. Another one 'repos.dcf' would be a good functional complementary to it. Those two simple helper functions really makes it easier for organizations to glue together usage of their own R packages repos and CRAN repo in a smooth way. That could possibly help to offload CRAN from new submissions. gh mirror link for easy preview: https://github.com/wch/r-source/blob/tools4pkgs/src/library/tools/R/packages.R#L419 Regards Jan Gorecki [[alternative HTML version deleted]]
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Hi Jan and Jan, Can you explain a little more what exactly you want the non-recursive, non-version aware dependencies from an individual package for? Either way package_dependencies will do this for you* with a little "aggressive convincing". It wants output from available.packages, but who really cares what it wants? It's a function and we are people :)
library(tools)
db <- read.dcf("~/gabe/checkedout/rtables_clean/DESCRIPTION")
package_dependencies("rtables", db, which = intersect(c("Depends",
"Suggests", "Imports", "LinkingTo"), colnames(db)))
$rtables
[1] "methods" "magrittr" "formatters" "dplyr" "tibble"
[6] "tidyr" "testthat" "xml2" "knitr" "rmarkdown"
[11] "flextable" "officer" "stats" "htmltools" "grid"
The only gotcha that I see immediately is that "LinkingTo" isn't always
there (whereas it is with real output from available.packages). If you know
your package doesn't have that (or that it does) at call time , this
becomes a one-liner:
package_dependencies("rtables", db =
read.dcf("~/gabe/checkedout/rtables_clean/DESCRIPTION"), which =
c("Depends", "Suggests", "Imports"))
$rtables
[1] "methods" "magrittr" "formatters" "dplyr" "tibble"
[6] "tidyr" "testthat" "xml2" "knitr" "rmarkdown"
[11] "flextable" "officer" "stats" "htmltools" "grid"
You can also trick it a slightly different way by giving it what it
actually wants
tdir <- tempdir()
file.copy("~/gabe/checkedout/rtables_clean/DESCRIPTION", file.path(tdir,
"PACKAGES")) [1] TRUE
avl <- available.packages(paste0("file://", tdir))
library(tools)
package_dependencies("rtables", avl)
$rtables [1] "methods" "magrittr" "formatters" "stats" "htmltools" [6] "grid"
package_dependencies("rtables", avl, which = "all")
$rtables [1] "methods" "magrittr" "formatters" "stats" "htmltools" [6] "grid" "dplyr" "tibble" "tidyr" "testthat" [11] "xml2" "knitr" "rmarkdown" "flextable" "officer" So the only real benefits I see that we'd be picking up here is automatic filtering by priority, and automatic extraction of the package name from the DESCRIPTION file. I'm not sure either of those warrant a new exported function that R-core has to maintain forever. Best, ~G * I haven't tested this across all OSes, but I dont' know of any reason it wouldn't work generally.
On Fri, Oct 14, 2022 at 2:33 PM Jan Gorecki <j.gorecki at wit.edu.pl> wrote:
Hello Jan, Thanks for confirming about many packages reinventing this missing functionality. packages.dcf was not meant handle versions. It just extracts names of dependencies... Yes, such a simple thing, yet missing in base R. Versions of packages can be controlled when setting up R pkgs repo. This is how I used to handle it. Making a CRAN subset mirror of fixed version pkgs. BTW. function for that is also included in mentioned branch. I am just not proposing it, to increase the chance of having at least this simple, missing, functionality merged. Best Jan On Fri, Oct 14, 2022, 15:14 Jan Net?k <netikja at gmail.com> wrote:
Hello Jan, I have seen many packages that implemented dependencies "extraction" on their own for internal purposes and today I was doing exactly that for mine. It's not a big deal using read.dcf on DESCRIPTION. It was
sufficient
for me, but I had to take care of some \n chars (the overall returned
value
has some rough edges, in my opinion). However, the function from the
branch
seems to not care about version requirements, which are crucial for me. Maybe that is something to reconsider before merging. Best, Jan p? 14. 10. 2022 v 2:27 odes?latel Jan Gorecki <j.gorecki at wit.edu.pl> napsal:
Dear R devs, I would like to raise a request for a simple helper function. Utility function to extract package dependencies from DESCRIPTION file. I do think that tools package is better place, for such a fundamental functionality, than community packages. tools pkg seems perfect fit (having already great function write_PACKAGES). Functionality I am asking for is already in R svn repository since 2016, in a branch tools4pkgs. Function is called 'packages.dcf'. Another one 'repos.dcf' would be a good functional complementary to it. Those two simple helper functions really makes it easier for
organizations
to glue together usage of their own R packages repos and CRAN repo in a smooth way. That could possibly help to offload CRAN from new
submissions.
gh mirror link for easy preview:
Regards
Jan Gorecki
[[alternative HTML version deleted]]
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
[[alternative HTML version deleted]]
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Hi Gabriel, It's very nice usage you provided here. Maybe instead of adding new function we could extend packages_depenedncies then? To accept file path to dsc file. What about repos.dcf? Maybe additional repositories could be an attribute attached to returned character vector. The use case is to, for a given package sources, obtain its dependencies, so one can use that for installing them/mirroring CRAN subset, or whatever. The later is especially important for a production environment where one wants to have fixed version of packages, and mirroring relevant subset of CRAN is the most simple, and IMO reliable, way to manage such environment. Regards Jan
On Fri, Oct 14, 2022, 23:34 Gabriel Becker <gabembecker at gmail.com> wrote:
Hi Jan and Jan, Can you explain a little more what exactly you want the non-recursive, non-version aware dependencies from an individual package for? Either way package_dependencies will do this for you* with a little "aggressive convincing". It wants output from available.packages, but who really cares what it wants? It's a function and we are people :)
library(tools)
db <- read.dcf("~/gabe/checkedout/rtables_clean/DESCRIPTION")
package_dependencies("rtables", db, which = intersect(c("Depends",
"Suggests", "Imports", "LinkingTo"), colnames(db)))
$rtables
[1] "methods" "magrittr" "formatters" "dplyr" "tibble"
[6] "tidyr" "testthat" "xml2" "knitr" "rmarkdown"
[11] "flextable" "officer" "stats" "htmltools" "grid"
The only gotcha that I see immediately is that "LinkingTo" isn't always
there (whereas it is with real output from available.packages). If you
know your package doesn't have that (or that it does) at call time , this
becomes a one-liner:
package_dependencies("rtables", db =
read.dcf("~/gabe/checkedout/rtables_clean/DESCRIPTION"), which =
c("Depends", "Suggests", "Imports"))
$rtables
[1] "methods" "magrittr" "formatters" "dplyr" "tibble"
[6] "tidyr" "testthat" "xml2" "knitr" "rmarkdown"
[11] "flextable" "officer" "stats" "htmltools" "grid"
You can also trick it a slightly different way by giving it what it
actually wants
tdir <- tempdir()
file.copy("~/gabe/checkedout/rtables_clean/DESCRIPTION", file.path(tdir,
"PACKAGES")) [1] TRUE
avl <- available.packages(paste0("file://", tdir))
library(tools)
package_dependencies("rtables", avl)
$rtables [1] "methods" "magrittr" "formatters" "stats" "htmltools" [6] "grid"
package_dependencies("rtables", avl, which = "all")
$rtables [1] "methods" "magrittr" "formatters" "stats" "htmltools" [6] "grid" "dplyr" "tibble" "tidyr" "testthat" [11] "xml2" "knitr" "rmarkdown" "flextable" "officer" So the only real benefits I see that we'd be picking up here is automatic filtering by priority, and automatic extraction of the package name from the DESCRIPTION file. I'm not sure either of those warrant a new exported function that R-core has to maintain forever. Best, ~G * I haven't tested this across all OSes, but I dont' know of any reason it wouldn't work generally. On Fri, Oct 14, 2022 at 2:33 PM Jan Gorecki <j.gorecki at wit.edu.pl> wrote:
Hello Jan, Thanks for confirming about many packages reinventing this missing functionality. packages.dcf was not meant handle versions. It just extracts names of dependencies... Yes, such a simple thing, yet missing in base R. Versions of packages can be controlled when setting up R pkgs repo. This is how I used to handle it. Making a CRAN subset mirror of fixed version pkgs. BTW. function for that is also included in mentioned branch. I am just not proposing it, to increase the chance of having at least this simple, missing, functionality merged. Best Jan On Fri, Oct 14, 2022, 15:14 Jan Net?k <netikja at gmail.com> wrote:
Hello Jan, I have seen many packages that implemented dependencies "extraction" on their own for internal purposes and today I was doing exactly that for mine. It's not a big deal using read.dcf on DESCRIPTION. It was
sufficient
for me, but I had to take care of some \n chars (the overall returned
value
has some rough edges, in my opinion). However, the function from the
branch
seems to not care about version requirements, which are crucial for me. Maybe that is something to reconsider before merging. Best, Jan p? 14. 10. 2022 v 2:27 odes?latel Jan Gorecki <j.gorecki at wit.edu.pl> napsal:
Dear R devs, I would like to raise a request for a simple helper function. Utility function to extract package dependencies from DESCRIPTION file. I do think that tools package is better place, for such a fundamental functionality, than community packages. tools pkg seems perfect fit (having already great function write_PACKAGES). Functionality I am asking for is already in R svn repository since
2016,
in a branch tools4pkgs. Function is called 'packages.dcf'. Another one 'repos.dcf' would be a good functional complementary to it. Those two simple helper functions really makes it easier for
organizations
to glue together usage of their own R packages repos and CRAN repo in a smooth way. That could possibly help to offload CRAN from new
submissions.
gh mirror link for easy preview:
Regards
Jan Gorecki
[[alternative HTML version deleted]]
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
[[alternative HTML version deleted]]
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
On Fri, Oct 14, 2022 at 11:14 PM Jan Gorecki <j.gorecki at wit.edu.pl> wrote:
Hi Gabriel, It's very nice usage you provided here. Maybe instead of adding new function we could extend packages_depenedncies then? To accept file path to dsc file. What about repos.dcf? Maybe additional repositories could be an attribute attached to returned character vector. The use case is to, for a given package sources, obtain its dependencies, so one can use that for installing them/mirroring CRAN subset, or whatever. The later is especially important for a production environment where one wants to have fixed version of packages, and mirroring relevant subset of CRAN is the most simple, and IMO reliable, way to manage such environment.
Right. Thats why I asked though, because this only makes sense to do recursively (i.e. collectively). Packages cannot meaningfully be treated in isolation in R. If you capture/mirror the non-recursive dependencies only, your package won't be (re-)installable. What you actually want is either a frozen slice of CRAN, or a description of either your full package library, or the full recursive subset of it relevant to a particular package. (switchr was designed to do both of these things easily, as an aside). Neither of which is achievable by looking at an individual DESCRIPTION file. Here's another fun trick if you don't want to just use switchr and let it take care of it for you:
.libPaths()
[1] "/Users/gabrielbecker/Rlib/syswide-4.1.2" [2] "/Library/Frameworks/R.framework/Versions/4.1/Resources/library"
write_PACKAGES(.libPaths()[1], unpacked = TRUE, validate = FALSE)
avl <- available.packages(paste0("file://", .libPaths()[1]))
head(avl)
Package Version Priority
abind "abind" "1.4-5" NA
AnnotationDbi "AnnotationDbi" "1.56.2" NA
askpass "askpass" "1.1" NA
assertthat "assertthat" "0.2.1" NA
backports "backports" "1.4.1" NA
base64enc "base64enc" "0.1-3" NA
Depends
abind "R (>= 1.5.0)"
AnnotationDbi "R (>= 2.7.0), methods, utils, stats4, BiocGenerics
(>=\n0.29.2), Biobase (>= 1.17.0), IRanges"
askpass NA
assertthat NA
backports "R (>= 3.0.0)"
base64enc "R (>= 2.9.0)"
Imports
LinkingTo
abind "methods, utils" NA
AnnotationDbi "DBI, RSQLite, S4Vectors (>= 0.9.25), stats, KEGGREST" NA
askpass "sys (>= 2.1)" NA
assertthat "tools" NA
backports NA NA
base64enc NA NA
Suggests
abind NA
AnnotationDbi "hgu95av2.db, GO.db, org.Sc.sgd.db, org.At.tair.db,
RUnit,\nTxDb.Hsapiens.UCSC.hg19.knownGene, org.Hs.eg.db,
reactome.db,\nAnnotationForge, graph, EnsDb.Hsapiens.v75, BiocStyle, knitr"
askpass "testthat"
assertthat "testthat, covr"
backports NA
base64enc NA
Enhances License License_is_FOSS
abind NA "LGPL (>= 2)" NA
AnnotationDbi NA "Artistic-2.0" NA
askpass NA "MIT + file LICENSE" NA
assertthat NA "GPL-3" NA
backports NA "GPL-2 | GPL-3" NA
base64enc "png" "GPL-2 | GPL-3" NA
License_restricts_use OS_type Archs MD5sum
abind NA NA NA NA
AnnotationDbi NA NA NA NA
askpass NA NA "askpass.so.dSYM" NA
assertthat NA NA NA NA
backports NA NA "backports.so.dSYM" NA
base64enc NA NA "base64enc.so.dSYM" NA
NeedsCompilation File
abind "no" NA
AnnotationDbi "no" NA
askpass "yes" NA
assertthat "no" NA
backports "yes" NA
base64enc "yes" NA
Repository
abind "file:///Users/gabrielbecker/Rlib/syswide-4.1.2"
AnnotationDbi "file:///Users/gabrielbecker/Rlib/syswide-4.1.2"
askpass "file:///Users/gabrielbecker/Rlib/syswide-4.1.2"
assertthat "file:///Users/gabrielbecker/Rlib/syswide-4.1.2"
backports "file:///Users/gabrielbecker/Rlib/syswide-4.1.2"
base64enc "file:///Users/gabrielbecker/Rlib/syswide-4.1.2"
package_dependencies("rtables", avl, recursive = TRUE)
$rtables [1] "methods" "magrittr" "formatters" "stats" "htmltools" [6] "grid" "utils" "digest" "grDevices" "base64enc" [11] "rlang" "fastmap"
package_dependencies("rtables", avl, which = "all", recursive = TRUE)
$rtables [1] "methods" [2] "magrittr" [3] "formatters" [4] "stats" [5] "htmltools" [6] "grid" [7] "dplyr" [8] "tibble" <snip> [653] "rjson" [654] "rsolr" [655] "rlecuyer" [656] "filelock" Now you should probably move the PACKAGES file somewhere else and not leave it in your package library, but I trust this illustrated my point. Most of the exported machinery is based on available.packages output, but that's not really a meaningful blocker for this type of work. We can get available.packages output if we need to. Remember the PACKAGES file is just a bunch of DESCRIPTION files slightly trimmed and then appended one after the other. This also shows why recursive and which=all don't really go together. In my opinion (and thus switchr's) the correct thing to do is do all for the package in question and then only hard dependencies of those packages recursively. That will let you build the package's vignettes (if you care about such things), but won't pull in hundreds or thousands of reverse deps. Best, ~G
Regards Jan On Fri, Oct 14, 2022, 23:34 Gabriel Becker <gabembecker at gmail.com> wrote:
Hi Jan and Jan, Can you explain a little more what exactly you want the non-recursive, non-version aware dependencies from an individual package for? Either way package_dependencies will do this for you* with a little "aggressive convincing". It wants output from available.packages, but who really cares what it wants? It's a function and we are people :)
library(tools)
db <- read.dcf("~/gabe/checkedout/rtables_clean/DESCRIPTION")
package_dependencies("rtables", db, which = intersect(c("Depends",
"Suggests", "Imports", "LinkingTo"), colnames(db)))
$rtables
[1] "methods" "magrittr" "formatters" "dplyr" "tibble"
[6] "tidyr" "testthat" "xml2" "knitr" "rmarkdown"
[11] "flextable" "officer" "stats" "htmltools" "grid"
The only gotcha that I see immediately is that "LinkingTo" isn't always
there (whereas it is with real output from available.packages). If you
know your package doesn't have that (or that it does) at call time , this
becomes a one-liner:
package_dependencies("rtables", db =
read.dcf("~/gabe/checkedout/rtables_clean/DESCRIPTION"), which =
c("Depends", "Suggests", "Imports"))
$rtables
[1] "methods" "magrittr" "formatters" "dplyr" "tibble"
[6] "tidyr" "testthat" "xml2" "knitr" "rmarkdown"
[11] "flextable" "officer" "stats" "htmltools" "grid"
You can also trick it a slightly different way by giving it what it
actually wants
tdir <- tempdir()
file.copy("~/gabe/checkedout/rtables_clean/DESCRIPTION",
file.path(tdir, "PACKAGES")) [1] TRUE
avl <- available.packages(paste0("file://", tdir))
library(tools)
package_dependencies("rtables", avl)
$rtables [1] "methods" "magrittr" "formatters" "stats" "htmltools" [6] "grid"
package_dependencies("rtables", avl, which = "all")
$rtables [1] "methods" "magrittr" "formatters" "stats" "htmltools" [6] "grid" "dplyr" "tibble" "tidyr" "testthat" [11] "xml2" "knitr" "rmarkdown" "flextable" "officer" So the only real benefits I see that we'd be picking up here is automatic filtering by priority, and automatic extraction of the package name from the DESCRIPTION file. I'm not sure either of those warrant a new exported function that R-core has to maintain forever. Best, ~G * I haven't tested this across all OSes, but I dont' know of any reason it wouldn't work generally. On Fri, Oct 14, 2022 at 2:33 PM Jan Gorecki <j.gorecki at wit.edu.pl> wrote:
Hello Jan, Thanks for confirming about many packages reinventing this missing functionality. packages.dcf was not meant handle versions. It just extracts names of dependencies... Yes, such a simple thing, yet missing in base R. Versions of packages can be controlled when setting up R pkgs repo. This is how I used to handle it. Making a CRAN subset mirror of fixed version pkgs. BTW. function for that is also included in mentioned branch. I am just not proposing it, to increase the chance of having at least this simple, missing, functionality merged. Best Jan On Fri, Oct 14, 2022, 15:14 Jan Net?k <netikja at gmail.com> wrote:
Hello Jan, I have seen many packages that implemented dependencies "extraction" on their own for internal purposes and today I was doing exactly that for mine. It's not a big deal using read.dcf on DESCRIPTION. It was
sufficient
for me, but I had to take care of some \n chars (the overall returned
value
has some rough edges, in my opinion). However, the function from the
branch
seems to not care about version requirements, which are crucial for me. Maybe that is something to reconsider before merging. Best, Jan p? 14. 10. 2022 v 2:27 odes?latel Jan Gorecki <j.gorecki at wit.edu.pl> napsal:
Dear R devs, I would like to raise a request for a simple helper function. Utility function to extract package dependencies from DESCRIPTION
file.
I do think that tools package is better place, for such a fundamental functionality, than community packages. tools pkg seems perfect fit (having already great function write_PACKAGES). Functionality I am asking for is already in R svn repository since
2016,
in a branch tools4pkgs. Function is called 'packages.dcf'. Another one 'repos.dcf' would be a good functional complementary to
it.
Those two simple helper functions really makes it easier for
organizations
to glue together usage of their own R packages repos and CRAN repo in
a
smooth way. That could possibly help to offload CRAN from new
submissions.
gh mirror link for easy preview:
Regards
Jan Gorecki
[[alternative HTML version deleted]]
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
[[alternative HTML version deleted]]
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Jan, I think using a single DCF as input is not very practical and would not be useful in the context you describe (creating self contained repos) since they typically concern a list of packages, but essentially splitting out the part of install.packages() which determines which files will be pulled from where would be very useful as it would be trivial to use it to create repository (what we always do in corporate environments) instead of installing the packages. I suspect that install packages is already too complex so instead of adding a flag to install.packages one could move that functionality into a separate function - we all do that constantly for the sites we manage, so it would be certainly something worthwhile. Cheers, Simon
On Oct 15, 2022, at 7:14 PM, Jan Gorecki <j.gorecki at wit.edu.pl> wrote: Hi Gabriel, It's very nice usage you provided here. Maybe instead of adding new function we could extend packages_depenedncies then? To accept file path to dsc file. What about repos.dcf? Maybe additional repositories could be an attribute attached to returned character vector. The use case is to, for a given package sources, obtain its dependencies, so one can use that for installing them/mirroring CRAN subset, or whatever. The later is especially important for a production environment where one wants to have fixed version of packages, and mirroring relevant subset of CRAN is the most simple, and IMO reliable, way to manage such environment. Regards Jan On Fri, Oct 14, 2022, 23:34 Gabriel Becker <gabembecker at gmail.com> wrote:
Hi Jan and Jan, Can you explain a little more what exactly you want the non-recursive, non-version aware dependencies from an individual package for? Either way package_dependencies will do this for you* with a little "aggressive convincing". It wants output from available.packages, but who really cares what it wants? It's a function and we are people :)
library(tools)
db <- read.dcf("~/gabe/checkedout/rtables_clean/DESCRIPTION")
package_dependencies("rtables", db, which = intersect(c("Depends",
"Suggests", "Imports", "LinkingTo"), colnames(db)))
$rtables
[1] "methods" "magrittr" "formatters" "dplyr" "tibble"
[6] "tidyr" "testthat" "xml2" "knitr" "rmarkdown"
[11] "flextable" "officer" "stats" "htmltools" "grid"
The only gotcha that I see immediately is that "LinkingTo" isn't always
there (whereas it is with real output from available.packages). If you
know your package doesn't have that (or that it does) at call time , this
becomes a one-liner:
package_dependencies("rtables", db =
read.dcf("~/gabe/checkedout/rtables_clean/DESCRIPTION"), which =
c("Depends", "Suggests", "Imports"))
$rtables
[1] "methods" "magrittr" "formatters" "dplyr" "tibble"
[6] "tidyr" "testthat" "xml2" "knitr" "rmarkdown"
[11] "flextable" "officer" "stats" "htmltools" "grid"
You can also trick it a slightly different way by giving it what it
actually wants
tdir <- tempdir()
file.copy("~/gabe/checkedout/rtables_clean/DESCRIPTION", file.path(tdir,
"PACKAGES")) [1] TRUE
avl <- available.packages(paste0("file://", tdir))
library(tools)
package_dependencies("rtables", avl)
$rtables [1] "methods" "magrittr" "formatters" "stats" "htmltools" [6] "grid"
package_dependencies("rtables", avl, which = "all")
$rtables [1] "methods" "magrittr" "formatters" "stats" "htmltools" [6] "grid" "dplyr" "tibble" "tidyr" "testthat" [11] "xml2" "knitr" "rmarkdown" "flextable" "officer" So the only real benefits I see that we'd be picking up here is automatic filtering by priority, and automatic extraction of the package name from the DESCRIPTION file. I'm not sure either of those warrant a new exported function that R-core has to maintain forever. Best, ~G * I haven't tested this across all OSes, but I dont' know of any reason it wouldn't work generally. On Fri, Oct 14, 2022 at 2:33 PM Jan Gorecki <j.gorecki at wit.edu.pl> wrote:
Hello Jan, Thanks for confirming about many packages reinventing this missing functionality. packages.dcf was not meant handle versions. It just extracts names of dependencies... Yes, such a simple thing, yet missing in base R. Versions of packages can be controlled when setting up R pkgs repo. This is how I used to handle it. Making a CRAN subset mirror of fixed version pkgs. BTW. function for that is also included in mentioned branch. I am just not proposing it, to increase the chance of having at least this simple, missing, functionality merged. Best Jan On Fri, Oct 14, 2022, 15:14 Jan Net?k <netikja at gmail.com> wrote:
Hello Jan, I have seen many packages that implemented dependencies "extraction" on their own for internal purposes and today I was doing exactly that for mine. It's not a big deal using read.dcf on DESCRIPTION. It was
sufficient
for me, but I had to take care of some \n chars (the overall returned
value
has some rough edges, in my opinion). However, the function from the
branch
seems to not care about version requirements, which are crucial for me. Maybe that is something to reconsider before merging. Best, Jan p? 14. 10. 2022 v 2:27 odes?latel Jan Gorecki <j.gorecki at wit.edu.pl> napsal:
Dear R devs, I would like to raise a request for a simple helper function. Utility function to extract package dependencies from DESCRIPTION file. I do think that tools package is better place, for such a fundamental functionality, than community packages. tools pkg seems perfect fit (having already great function write_PACKAGES). Functionality I am asking for is already in R svn repository since
2016,
in a branch tools4pkgs. Function is called 'packages.dcf'. Another one 'repos.dcf' would be a good functional complementary to it. Those two simple helper functions really makes it easier for
organizations
to glue together usage of their own R packages repos and CRAN repo in a smooth way. That could possibly help to offload CRAN from new
submissions.
gh mirror link for easy preview:
Regards
Jan Gorecki
[[alternative HTML version deleted]]
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
[[alternative HTML version deleted]]
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
[[alternative HTML version deleted]]
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
1 day later
Gabriel and Simon I completely agree with what you are saying. The thing is that obtaining recursive deps, all/most whatever, is already well supported in core R. What is missing is just this single functionality I am requesting. If you will look into the branch you can see there is mirror.packages function meant to mirror a slice of CRAN. It is doing exactly what you described: package_dependencies; to obtain recursive deps, then download all, etc. I would love to have this function provided by core R as well, but we need to start somewhere. There are other use cases as well. For example CI, where one wants to install all/most dependencies and then run R CMD check. Then we don't worry about recursive deps are they will be resolved automatically. I don't think it's reasonable to force users to use 3rd party packages to handle such a common and simple use case. Otherwise one has to hard code deps in CI script. Not robust at all. packages.dcf and repos.dcf makes all that way easier, and are solid base for building customized orchestration like mirroring slice of CRAN. Best regards Jan On Sun, Oct 16, 2022, 01:31 Simon Urbanek <simon.urbanek at r-project.org> wrote:
Jan, I think using a single DCF as input is not very practical and would not be useful in the context you describe (creating self contained repos) since they typically concern a list of packages, but essentially splitting out the part of install.packages() which determines which files will be pulled from where would be very useful as it would be trivial to use it to create repository (what we always do in corporate environments) instead of installing the packages. I suspect that install packages is already too complex so instead of adding a flag to install.packages one could move that functionality into a separate function - we all do that constantly for the sites we manage, so it would be certainly something worthwhile. Cheers, Simon
On Oct 15, 2022, at 7:14 PM, Jan Gorecki <j.gorecki at wit.edu.pl> wrote: Hi Gabriel, It's very nice usage you provided here. Maybe instead of adding new function we could extend packages_depenedncies then? To accept file path
to
dsc file. What about repos.dcf? Maybe additional repositories could be an attribute attached to returned character vector. The use case is to, for a given package sources, obtain its dependencies, so one can use that for installing them/mirroring CRAN subset, or
whatever.
The later is especially important for a production environment where one wants to have fixed version of packages, and mirroring relevant subset of CRAN is the most simple, and IMO reliable, way to manage such
environment.
Regards Jan On Fri, Oct 14, 2022, 23:34 Gabriel Becker <gabembecker at gmail.com>
wrote:
Hi Jan and Jan, Can you explain a little more what exactly you want the non-recursive, non-version aware dependencies from an individual package for? Either way package_dependencies will do this for you* with a little "aggressive convincing". It wants output from available.packages, but
who
really cares what it wants? It's a function and we are people :)
library(tools)
db <- read.dcf("~/gabe/checkedout/rtables_clean/DESCRIPTION")
package_dependencies("rtables", db, which = intersect(c("Depends",
"Suggests", "Imports", "LinkingTo"), colnames(db))) $rtables [1] "methods" "magrittr" "formatters" "dplyr" "tibble" [6] "tidyr" "testthat" "xml2" "knitr" "rmarkdown" [11] "flextable" "officer" "stats" "htmltools" "grid" The only gotcha that I see immediately is that "LinkingTo" isn't always there (whereas it is with real output from available.packages). If you know your package doesn't have that (or that it does) at call time ,
this
becomes a one-liner:
package_dependencies("rtables", db =
read.dcf("~/gabe/checkedout/rtables_clean/DESCRIPTION"), which =
c("Depends", "Suggests", "Imports"))
$rtables
[1] "methods" "magrittr" "formatters" "dplyr" "tibble"
[6] "tidyr" "testthat" "xml2" "knitr" "rmarkdown"
[11] "flextable" "officer" "stats" "htmltools" "grid"
You can also trick it a slightly different way by giving it what it
actually wants
tdir <- tempdir()
file.copy("~/gabe/checkedout/rtables_clean/DESCRIPTION",
file.path(tdir,
"PACKAGES")) [1] TRUE
avl <- available.packages(paste0("file://", tdir))
library(tools)
package_dependencies("rtables", avl)
$rtables [1] "methods" "magrittr" "formatters" "stats" "htmltools" [6] "grid"
package_dependencies("rtables", avl, which = "all")
$rtables [1] "methods" "magrittr" "formatters" "stats" "htmltools" [6] "grid" "dplyr" "tibble" "tidyr" "testthat" [11] "xml2" "knitr" "rmarkdown" "flextable" "officer" So the only real benefits I see that we'd be picking up here is
automatic
filtering by priority, and automatic extraction of the package name from the DESCRIPTION file. I'm not sure either of those warrant a new
exported
function that R-core has to maintain forever. Best, ~G * I haven't tested this across all OSes, but I dont' know of any reason
it
wouldn't work generally. On Fri, Oct 14, 2022 at 2:33 PM Jan Gorecki <j.gorecki at wit.edu.pl>
wrote:
Hello Jan, Thanks for confirming about many packages reinventing this missing functionality. packages.dcf was not meant handle versions. It just extracts names of dependencies... Yes, such a simple thing, yet missing in base R. Versions of packages can be controlled when setting up R pkgs repo.
This
is how I used to handle it. Making a CRAN subset mirror of fixed version pkgs. BTW. function for that is also included in mentioned branch. I am just
not
proposing it, to increase the chance of having at least this simple, missing, functionality merged. Best Jan On Fri, Oct 14, 2022, 15:14 Jan Net?k <netikja at gmail.com> wrote:
Hello Jan, I have seen many packages that implemented dependencies "extraction"
on
their own for internal purposes and today I was doing exactly that for mine. It's not a big deal using read.dcf on DESCRIPTION. It was
sufficient
for me, but I had to take care of some \n chars (the overall returned
value
has some rough edges, in my opinion). However, the function from the
branch
seems to not care about version requirements, which are crucial for
me.
Maybe that is something to reconsider before merging. Best, Jan p? 14. 10. 2022 v 2:27 odes?latel Jan Gorecki <j.gorecki at wit.edu.pl> napsal:
Dear R devs, I would like to raise a request for a simple helper function. Utility function to extract package dependencies from DESCRIPTION
file.
I do think that tools package is better place, for such a fundamental functionality, than community packages. tools pkg seems perfect fit (having already great function write_PACKAGES). Functionality I am asking for is already in R svn repository since
2016,
in a branch tools4pkgs. Function is called 'packages.dcf'. Another one 'repos.dcf' would be a good functional complementary to
it.
Those two simple helper functions really makes it easier for
organizations
to glue together usage of their own R packages repos and CRAN repo
in a
smooth way. That could possibly help to offload CRAN from new
submissions.
gh mirror link for easy preview:
Regards
Jan Gorecki
[[alternative HTML version deleted]]
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
[[alternative HTML version deleted]]
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
[[alternative HTML version deleted]]
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
11 days later
Gabriel,
I am trying to design generic solution that could be applied to
arbitrary package. Therefore I went with the latter solution you
proposed.
If we wouldn't have to exclude base packages, then its a 3 liner
file.copy("DESCRIPTION", file.path(tdir<-tempdir(), "PACKAGES"));
db<-available.packages(paste0("file://", tdir));
utils::install.packages(tools::package_dependencies("pkgname", db,
which="most")[[1L]])
As you noticed, we still have to filter out base packages. Otherwise
it won't be a robust utility that can be used in CI. Therefore we have
to add a call to tools:::.get_standard_package_names() which is an
internal function (as of now). Not only complicating the call but also
putting the functionality outside of safe use.
Considering above, don't you agree that the following one liner could
nicely address the problem? The problem that hundreds/thousands of
packages are now addressing in their CI scripts by using a third party
packages.
utils::install.packages(packages.dcf("DESCRIPTION", which="most"))
It is hard to me to understand why R members don't consider this basic
functionality to be part of base R. Possibly they just don't need it
themselves. Yet isn't this sufficient that hundreds/thousands of
packages does need this functionality?
Best regards,
Jan
On Mon, Oct 17, 2022 at 8:39 AM Jan Gorecki <j.gorecki at wit.edu.pl> wrote:
Gabriel and Simon I completely agree with what you are saying. The thing is that obtaining recursive deps, all/most whatever, is already well supported in core R. What is missing is just this single functionality I am requesting. If you will look into the branch you can see there is mirror.packages function meant to mirror a slice of CRAN. It is doing exactly what you described: package_dependencies; to obtain recursive deps, then download all, etc. I would love to have this function provided by core R as well, but we need to start somewhere. There are other use cases as well. For example CI, where one wants to install all/most dependencies and then run R CMD check. Then we don't worry about recursive deps are they will be resolved automatically. I don't think it's reasonable to force users to use 3rd party packages to handle such a common and simple use case. Otherwise one has to hard code deps in CI script. Not robust at all. packages.dcf and repos.dcf makes all that way easier, and are solid base for building customized orchestration like mirroring slice of CRAN. Best regards Jan On Sun, Oct 16, 2022, 01:31 Simon Urbanek <simon.urbanek at r-project.org> wrote:
Jan, I think using a single DCF as input is not very practical and would not be useful in the context you describe (creating self contained repos) since they typically concern a list of packages, but essentially splitting out the part of install.packages() which determines which files will be pulled from where would be very useful as it would be trivial to use it to create repository (what we always do in corporate environments) instead of installing the packages. I suspect that install packages is already too complex so instead of adding a flag to install.packages one could move that functionality into a separate function - we all do that constantly for the sites we manage, so it would be certainly something worthwhile. Cheers, Simon
On Oct 15, 2022, at 7:14 PM, Jan Gorecki <j.gorecki at wit.edu.pl> wrote: Hi Gabriel, It's very nice usage you provided here. Maybe instead of adding new function we could extend packages_depenedncies then? To accept file path to dsc file. What about repos.dcf? Maybe additional repositories could be an attribute attached to returned character vector. The use case is to, for a given package sources, obtain its dependencies, so one can use that for installing them/mirroring CRAN subset, or whatever. The later is especially important for a production environment where one wants to have fixed version of packages, and mirroring relevant subset of CRAN is the most simple, and IMO reliable, way to manage such environment. Regards Jan On Fri, Oct 14, 2022, 23:34 Gabriel Becker <gabembecker at gmail.com> wrote:
Hi Jan and Jan, Can you explain a little more what exactly you want the non-recursive, non-version aware dependencies from an individual package for? Either way package_dependencies will do this for you* with a little "aggressive convincing". It wants output from available.packages, but who really cares what it wants? It's a function and we are people :)
library(tools)
db <- read.dcf("~/gabe/checkedout/rtables_clean/DESCRIPTION")
package_dependencies("rtables", db, which = intersect(c("Depends",
"Suggests", "Imports", "LinkingTo"), colnames(db)))
$rtables
[1] "methods" "magrittr" "formatters" "dplyr" "tibble"
[6] "tidyr" "testthat" "xml2" "knitr" "rmarkdown"
[11] "flextable" "officer" "stats" "htmltools" "grid"
The only gotcha that I see immediately is that "LinkingTo" isn't always
there (whereas it is with real output from available.packages). If you
know your package doesn't have that (or that it does) at call time , this
becomes a one-liner:
package_dependencies("rtables", db =
read.dcf("~/gabe/checkedout/rtables_clean/DESCRIPTION"), which =
c("Depends", "Suggests", "Imports"))
$rtables
[1] "methods" "magrittr" "formatters" "dplyr" "tibble"
[6] "tidyr" "testthat" "xml2" "knitr" "rmarkdown"
[11] "flextable" "officer" "stats" "htmltools" "grid"
You can also trick it a slightly different way by giving it what it
actually wants
tdir <- tempdir()
file.copy("~/gabe/checkedout/rtables_clean/DESCRIPTION", file.path(tdir,
"PACKAGES")) [1] TRUE
avl <- available.packages(paste0("file://", tdir))
library(tools)
package_dependencies("rtables", avl)
$rtables [1] "methods" "magrittr" "formatters" "stats" "htmltools" [6] "grid"
package_dependencies("rtables", avl, which = "all")
$rtables [1] "methods" "magrittr" "formatters" "stats" "htmltools" [6] "grid" "dplyr" "tibble" "tidyr" "testthat" [11] "xml2" "knitr" "rmarkdown" "flextable" "officer" So the only real benefits I see that we'd be picking up here is automatic filtering by priority, and automatic extraction of the package name from the DESCRIPTION file. I'm not sure either of those warrant a new exported function that R-core has to maintain forever. Best, ~G * I haven't tested this across all OSes, but I dont' know of any reason it wouldn't work generally. On Fri, Oct 14, 2022 at 2:33 PM Jan Gorecki <j.gorecki at wit.edu.pl> wrote:
Hello Jan, Thanks for confirming about many packages reinventing this missing functionality. packages.dcf was not meant handle versions. It just extracts names of dependencies... Yes, such a simple thing, yet missing in base R. Versions of packages can be controlled when setting up R pkgs repo. This is how I used to handle it. Making a CRAN subset mirror of fixed version pkgs. BTW. function for that is also included in mentioned branch. I am just not proposing it, to increase the chance of having at least this simple, missing, functionality merged. Best Jan On Fri, Oct 14, 2022, 15:14 Jan Net?k <netikja at gmail.com> wrote:
Hello Jan, I have seen many packages that implemented dependencies "extraction" on their own for internal purposes and today I was doing exactly that for mine. It's not a big deal using read.dcf on DESCRIPTION. It was
sufficient
for me, but I had to take care of some \n chars (the overall returned
value
has some rough edges, in my opinion). However, the function from the
branch
seems to not care about version requirements, which are crucial for me. Maybe that is something to reconsider before merging. Best, Jan p? 14. 10. 2022 v 2:27 odes?latel Jan Gorecki <j.gorecki at wit.edu.pl> napsal:
Dear R devs, I would like to raise a request for a simple helper function. Utility function to extract package dependencies from DESCRIPTION file. I do think that tools package is better place, for such a fundamental functionality, than community packages. tools pkg seems perfect fit (having already great function write_PACKAGES). Functionality I am asking for is already in R svn repository since
2016,
in a branch tools4pkgs. Function is called 'packages.dcf'. Another one 'repos.dcf' would be a good functional complementary to it. Those two simple helper functions really makes it easier for
organizations
to glue together usage of their own R packages repos and CRAN repo in a smooth way. That could possibly help to offload CRAN from new
submissions.
gh mirror link for easy preview:
Regards
Jan Gorecki
[[alternative HTML version deleted]]
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
[[alternative HTML version deleted]]
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
[[alternative HTML version deleted]]
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Hi Jan, The reason, I suspect without speaking for R-core, is that by design you should not be specifying package dependencies as additional packages to install. install.packages already does this for you, as it did in the construct of a repository code that I provided previously in the thread. You should be *only* doing install.packages(<pkg in question>, repos = *) Then everything happens automatically via extremely well tested very mature code. I (still) don't understand why you'd need to pass install.packages the vector of dependencies yourself, as that is counter to install.packages' core design. Does that make sense? Best, ~G
On Fri, Oct 28, 2022 at 12:18 PM Jan Gorecki <j.gorecki at wit.edu.pl> wrote:
Gabriel,
I am trying to design generic solution that could be applied to
arbitrary package. Therefore I went with the latter solution you
proposed.
If we wouldn't have to exclude base packages, then its a 3 liner
file.copy("DESCRIPTION", file.path(tdir<-tempdir(), "PACKAGES"));
db<-available.packages(paste0("file://", tdir));
utils::install.packages(tools::package_dependencies("pkgname", db,
which="most")[[1L]])
As you noticed, we still have to filter out base packages. Otherwise
it won't be a robust utility that can be used in CI. Therefore we have
to add a call to tools:::.get_standard_package_names() which is an
internal function (as of now). Not only complicating the call but also
putting the functionality outside of safe use.
Considering above, don't you agree that the following one liner could
nicely address the problem? The problem that hundreds/thousands of
packages are now addressing in their CI scripts by using a third party
packages.
utils::install.packages(packages.dcf("DESCRIPTION", which="most"))
It is hard to me to understand why R members don't consider this basic
functionality to be part of base R. Possibly they just don't need it
themselves. Yet isn't this sufficient that hundreds/thousands of
packages does need this functionality?
Best regards,
Jan
On Mon, Oct 17, 2022 at 8:39 AM Jan Gorecki <j.gorecki at wit.edu.pl> wrote:
Gabriel and Simon I completely agree with what you are saying. The thing is that obtaining recursive deps, all/most whatever, is
already well supported in core R. What is missing is just this single functionality I am requesting.
If you will look into the branch you can see there is mirror.packages
function meant to mirror a slice of CRAN. It is doing exactly what you described: package_dependencies; to obtain recursive deps, then download all, etc.
I would love to have this function provided by core R as well, but we
need to start somewhere.
There are other use cases as well. For example CI, where one wants to install all/most dependencies and
then run R CMD check. Then we don't worry about recursive deps are they will be resolved automatically.
I don't think it's reasonable to force users to use 3rd party packages
to handle such a common and simple use case. Otherwise one has to hard code deps in CI script. Not robust at all.
packages.dcf and repos.dcf makes all that way easier, and are solid base
for building customized orchestration like mirroring slice of CRAN.
Best regards Jan On Sun, Oct 16, 2022, 01:31 Simon Urbanek <simon.urbanek at r-project.org>
wrote:
Jan, I think using a single DCF as input is not very practical and would not
be useful in the context you describe (creating self contained repos) since they typically concern a list of packages, but essentially splitting out the part of install.packages() which determines which files will be pulled from where would be very useful as it would be trivial to use it to create repository (what we always do in corporate environments) instead of installing the packages. I suspect that install packages is already too complex so instead of adding a flag to install.packages one could move that functionality into a separate function - we all do that constantly for the sites we manage, so it would be certainly something worthwhile.
Cheers, Simon
On Oct 15, 2022, at 7:14 PM, Jan Gorecki <j.gorecki at wit.edu.pl>
wrote:
Hi Gabriel, It's very nice usage you provided here. Maybe instead of adding new function we could extend packages_depenedncies then? To accept file
path to
dsc file. What about repos.dcf? Maybe additional repositories could be an
attribute
attached to returned character vector. The use case is to, for a given package sources, obtain its
dependencies,
so one can use that for installing them/mirroring CRAN subset, or
whatever.
The later is especially important for a production environment where
one
wants to have fixed version of packages, and mirroring relevant
subset of
CRAN is the most simple, and IMO reliable, way to manage such
environment.
Regards Jan On Fri, Oct 14, 2022, 23:34 Gabriel Becker <gabembecker at gmail.com>
wrote:
Hi Jan and Jan, Can you explain a little more what exactly you want the
non-recursive,
non-version aware dependencies from an individual package for? Either way package_dependencies will do this for you* with a little "aggressive convincing". It wants output from available.packages,
but who
really cares what it wants? It's a function and we are people :)
library(tools)
db <- read.dcf("~/gabe/checkedout/rtables_clean/DESCRIPTION")
package_dependencies("rtables", db, which = intersect(c("Depends",
"Suggests", "Imports", "LinkingTo"), colnames(db))) $rtables [1] "methods" "magrittr" "formatters" "dplyr" "tibble" [6] "tidyr" "testthat" "xml2" "knitr" "rmarkdown" [11] "flextable" "officer" "stats" "htmltools" "grid" The only gotcha that I see immediately is that "LinkingTo" isn't
always
there (whereas it is with real output from available.packages). If
you
know your package doesn't have that (or that it does) at call time ,
this
becomes a one-liner:
package_dependencies("rtables", db =
read.dcf("~/gabe/checkedout/rtables_clean/DESCRIPTION"), which =
c("Depends", "Suggests", "Imports"))
$rtables
[1] "methods" "magrittr" "formatters" "dplyr" "tibble"
[6] "tidyr" "testthat" "xml2" "knitr" "rmarkdown"
[11] "flextable" "officer" "stats" "htmltools" "grid"
You can also trick it a slightly different way by giving it what it
actually wants
tdir <- tempdir()
file.copy("~/gabe/checkedout/rtables_clean/DESCRIPTION",
file.path(tdir,
"PACKAGES")) [1] TRUE
avl <- available.packages(paste0("file://", tdir))
library(tools)
package_dependencies("rtables", avl)
$rtables [1] "methods" "magrittr" "formatters" "stats" "htmltools" [6] "grid"
package_dependencies("rtables", avl, which = "all")
$rtables [1] "methods" "magrittr" "formatters" "stats" "htmltools" [6] "grid" "dplyr" "tibble" "tidyr" "testthat" [11] "xml2" "knitr" "rmarkdown" "flextable" "officer" So the only real benefits I see that we'd be picking up here is
automatic
filtering by priority, and automatic extraction of the package name
from
the DESCRIPTION file. I'm not sure either of those warrant a new
exported
function that R-core has to maintain forever. Best, ~G * I haven't tested this across all OSes, but I dont' know of any
reason it
wouldn't work generally. On Fri, Oct 14, 2022 at 2:33 PM Jan Gorecki <j.gorecki at wit.edu.pl>
wrote:
Hello Jan, Thanks for confirming about many packages reinventing this missing functionality. packages.dcf was not meant handle versions. It just extracts names
of
dependencies... Yes, such a simple thing, yet missing in base R. Versions of packages can be controlled when setting up R pkgs repo.
This
is how I used to handle it. Making a CRAN subset mirror of fixed
version
pkgs. BTW. function for that is also included in mentioned branch. I am
just not
proposing it, to increase the chance of having at least this simple, missing, functionality merged. Best Jan On Fri, Oct 14, 2022, 15:14 Jan Net?k <netikja at gmail.com> wrote:
Hello Jan, I have seen many packages that implemented dependencies
"extraction" on
their own for internal purposes and today I was doing exactly that
for
mine. It's not a big deal using read.dcf on DESCRIPTION. It was
sufficient
for me, but I had to take care of some \n chars (the overall
returned
value
has some rough edges, in my opinion). However, the function from
the
branch
seems to not care about version requirements, which are crucial
for me.
Maybe that is something to reconsider before merging. Best, Jan p? 14. 10. 2022 v 2:27 odes?latel Jan Gorecki <
j.gorecki at wit.edu.pl>
napsal:
Dear R devs, I would like to raise a request for a simple helper function. Utility function to extract package dependencies from DESCRIPTION
file.
I do think that tools package is better place, for such a
fundamental
functionality, than community packages. tools pkg seems perfect fit (having already great function write_PACKAGES). Functionality I am asking for is already in R svn repository since
2016,
in a branch tools4pkgs. Function is called 'packages.dcf'. Another one 'repos.dcf' would be a good functional complementary
to it.
Those two simple helper functions really makes it easier for
organizations
to glue together usage of their own R packages repos and CRAN
repo in a
smooth way. That could possibly help to offload CRAN from new
submissions.
gh mirror link for easy preview:
Regards
Jan Gorecki
[[alternative HTML version deleted]]
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
[[alternative HTML version deleted]]
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
[[alternative HTML version deleted]]
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Gabriel, It is the most basic CI use case. One wants to install only dependencies only of the package, and run R CMD check on the package. Unless you say that installing the package and then running R CMD check on that package is considered good practice. Then yes, functionality I am asking about is not needed. Somehow I never thought that this could be considered a good practice just by the fact that installation of the package could already impact environment in which check is taking place. Best, Jan
On Fri, Oct 28, 2022 at 8:42 PM Gabriel Becker <gabembecker at gmail.com> wrote:
Hi Jan, The reason, I suspect without speaking for R-core, is that by design you should not be specifying package dependencies as additional packages to install. install.packages already does this for you, as it did in the construct of a repository code that I provided previously in the thread. You should be *only* doing install.packages(<pkg in question>, repos = *) Then everything happens automatically via extremely well tested very mature code. I (still) don't understand why you'd need to pass install.packages the vector of dependencies yourself, as that is counter to install.packages' core design. Does that make sense? Best, ~G On Fri, Oct 28, 2022 at 12:18 PM Jan Gorecki <j.gorecki at wit.edu.pl> wrote:
Gabriel,
I am trying to design generic solution that could be applied to
arbitrary package. Therefore I went with the latter solution you
proposed.
If we wouldn't have to exclude base packages, then its a 3 liner
file.copy("DESCRIPTION", file.path(tdir<-tempdir(), "PACKAGES"));
db<-available.packages(paste0("file://", tdir));
utils::install.packages(tools::package_dependencies("pkgname", db,
which="most")[[1L]])
As you noticed, we still have to filter out base packages. Otherwise
it won't be a robust utility that can be used in CI. Therefore we have
to add a call to tools:::.get_standard_package_names() which is an
internal function (as of now). Not only complicating the call but also
putting the functionality outside of safe use.
Considering above, don't you agree that the following one liner could
nicely address the problem? The problem that hundreds/thousands of
packages are now addressing in their CI scripts by using a third party
packages.
utils::install.packages(packages.dcf("DESCRIPTION", which="most"))
It is hard to me to understand why R members don't consider this basic
functionality to be part of base R. Possibly they just don't need it
themselves. Yet isn't this sufficient that hundreds/thousands of
packages does need this functionality?
Best regards,
Jan
On Mon, Oct 17, 2022 at 8:39 AM Jan Gorecki <j.gorecki at wit.edu.pl> wrote:
Gabriel and Simon I completely agree with what you are saying. The thing is that obtaining recursive deps, all/most whatever, is already well supported in core R. What is missing is just this single functionality I am requesting. If you will look into the branch you can see there is mirror.packages function meant to mirror a slice of CRAN. It is doing exactly what you described: package_dependencies; to obtain recursive deps, then download all, etc. I would love to have this function provided by core R as well, but we need to start somewhere. There are other use cases as well. For example CI, where one wants to install all/most dependencies and then run R CMD check. Then we don't worry about recursive deps are they will be resolved automatically. I don't think it's reasonable to force users to use 3rd party packages to handle such a common and simple use case. Otherwise one has to hard code deps in CI script. Not robust at all. packages.dcf and repos.dcf makes all that way easier, and are solid base for building customized orchestration like mirroring slice of CRAN. Best regards Jan On Sun, Oct 16, 2022, 01:31 Simon Urbanek <simon.urbanek at r-project.org> wrote:
Jan, I think using a single DCF as input is not very practical and would not be useful in the context you describe (creating self contained repos) since they typically concern a list of packages, but essentially splitting out the part of install.packages() which determines which files will be pulled from where would be very useful as it would be trivial to use it to create repository (what we always do in corporate environments) instead of installing the packages. I suspect that install packages is already too complex so instead of adding a flag to install.packages one could move that functionality into a separate function - we all do that constantly for the sites we manage, so it would be certainly something worthwhile. Cheers, Simon
On Oct 15, 2022, at 7:14 PM, Jan Gorecki <j.gorecki at wit.edu.pl> wrote: Hi Gabriel, It's very nice usage you provided here. Maybe instead of adding new function we could extend packages_depenedncies then? To accept file path to dsc file. What about repos.dcf? Maybe additional repositories could be an attribute attached to returned character vector. The use case is to, for a given package sources, obtain its dependencies, so one can use that for installing them/mirroring CRAN subset, or whatever. The later is especially important for a production environment where one wants to have fixed version of packages, and mirroring relevant subset of CRAN is the most simple, and IMO reliable, way to manage such environment. Regards Jan On Fri, Oct 14, 2022, 23:34 Gabriel Becker <gabembecker at gmail.com> wrote:
Hi Jan and Jan, Can you explain a little more what exactly you want the non-recursive, non-version aware dependencies from an individual package for? Either way package_dependencies will do this for you* with a little "aggressive convincing". It wants output from available.packages, but who really cares what it wants? It's a function and we are people :)
library(tools)
db <- read.dcf("~/gabe/checkedout/rtables_clean/DESCRIPTION")
package_dependencies("rtables", db, which = intersect(c("Depends",
"Suggests", "Imports", "LinkingTo"), colnames(db)))
$rtables
[1] "methods" "magrittr" "formatters" "dplyr" "tibble"
[6] "tidyr" "testthat" "xml2" "knitr" "rmarkdown"
[11] "flextable" "officer" "stats" "htmltools" "grid"
The only gotcha that I see immediately is that "LinkingTo" isn't always
there (whereas it is with real output from available.packages). If you
know your package doesn't have that (or that it does) at call time , this
becomes a one-liner:
package_dependencies("rtables", db =
read.dcf("~/gabe/checkedout/rtables_clean/DESCRIPTION"), which =
c("Depends", "Suggests", "Imports"))
$rtables
[1] "methods" "magrittr" "formatters" "dplyr" "tibble"
[6] "tidyr" "testthat" "xml2" "knitr" "rmarkdown"
[11] "flextable" "officer" "stats" "htmltools" "grid"
You can also trick it a slightly different way by giving it what it
actually wants
tdir <- tempdir()
file.copy("~/gabe/checkedout/rtables_clean/DESCRIPTION", file.path(tdir,
"PACKAGES")) [1] TRUE
avl <- available.packages(paste0("file://", tdir))
library(tools)
package_dependencies("rtables", avl)
$rtables [1] "methods" "magrittr" "formatters" "stats" "htmltools" [6] "grid"
package_dependencies("rtables", avl, which = "all")
$rtables [1] "methods" "magrittr" "formatters" "stats" "htmltools" [6] "grid" "dplyr" "tibble" "tidyr" "testthat" [11] "xml2" "knitr" "rmarkdown" "flextable" "officer" So the only real benefits I see that we'd be picking up here is automatic filtering by priority, and automatic extraction of the package name from the DESCRIPTION file. I'm not sure either of those warrant a new exported function that R-core has to maintain forever. Best, ~G * I haven't tested this across all OSes, but I dont' know of any reason it wouldn't work generally. On Fri, Oct 14, 2022 at 2:33 PM Jan Gorecki <j.gorecki at wit.edu.pl> wrote:
Hello Jan, Thanks for confirming about many packages reinventing this missing functionality. packages.dcf was not meant handle versions. It just extracts names of dependencies... Yes, such a simple thing, yet missing in base R. Versions of packages can be controlled when setting up R pkgs repo. This is how I used to handle it. Making a CRAN subset mirror of fixed version pkgs. BTW. function for that is also included in mentioned branch. I am just not proposing it, to increase the chance of having at least this simple, missing, functionality merged. Best Jan On Fri, Oct 14, 2022, 15:14 Jan Net?k <netikja at gmail.com> wrote:
Hello Jan, I have seen many packages that implemented dependencies "extraction" on their own for internal purposes and today I was doing exactly that for mine. It's not a big deal using read.dcf on DESCRIPTION. It was
sufficient
for me, but I had to take care of some \n chars (the overall returned
value
has some rough edges, in my opinion). However, the function from the
branch
seems to not care about version requirements, which are crucial for me. Maybe that is something to reconsider before merging. Best, Jan p? 14. 10. 2022 v 2:27 odes?latel Jan Gorecki <j.gorecki at wit.edu.pl> napsal:
Dear R devs, I would like to raise a request for a simple helper function. Utility function to extract package dependencies from DESCRIPTION file. I do think that tools package is better place, for such a fundamental functionality, than community packages. tools pkg seems perfect fit (having already great function write_PACKAGES). Functionality I am asking for is already in R svn repository since
2016,
in a branch tools4pkgs. Function is called 'packages.dcf'. Another one 'repos.dcf' would be a good functional complementary to it. Those two simple helper functions really makes it easier for
organizations
to glue together usage of their own R packages repos and CRAN repo in a smooth way. That could possibly help to offload CRAN from new
submissions.
gh mirror link for easy preview:
Regards
Jan Gorecki
[[alternative HTML version deleted]]
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
[[alternative HTML version deleted]]
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
[[alternative HTML version deleted]]
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Hi Jan,
On Fri, Oct 28, 2022 at 1:57 PM Jan Gorecki <j.gorecki at wit.edu.pl> wrote:
Gabriel, It is the most basic CI use case. One wants to install only dependencies only of the package, and run R CMD check on the package.
Really what you're looking for though, is to install all the dependencies
which aren't present right? Excluding base packages is just a particular
way to do that under certain assumptions about the CI environment.
So
needed_pkgs <- setdiff(package_dependencies(...),
installed.packages()[,"Package"])
install.packages(needed_pkgs, repos = fancyrepos)
will do what you want without installing the package itself, if that is
important. This will filter out base and recommended packages (which will
be already installed in your CI container, since R is).
Now this does not take into account versioned dependencies, so it's not
actually fully correct (whereas installing the package is), but it gets you
where you're trying to go. And in a clean CI container without cached
package installation for the deps, its equivalent.
Also, as an aside, if you need to get the base packages, you can do
installed.packages(priority="base")[,"Package"]
base compiler datasets graphics grDevices grid
"base" "compiler" "datasets" "graphics" "grDevices" "grid"
methods parallel splines stats stats4 tcltk
"methods" "parallel" "splines" "stats" "stats4" "tcltk"
tools utils
"tools" "utils"
(to get base and recommended packages use 'high' instead of 'base')
No need to be reaching down into unexported functions. So if you *really*
only want to exclude base functions (which likely will give you some
protection from versioned dep issues), you can change the code above to
needed_pkgs <- setdiff(package_dependencies(...),
installed.packages(priority = "high")[,"Package"])
install.packages(needed_pkgs, repos = fancyrepos)
Best,
~G
On Fri, Oct 28, 2022 at 8:42 PM Gabriel Becker <gabembecker at gmail.com> wrote:
Hi Jan, The reason, I suspect without speaking for R-core, is that by design you
should not be specifying package dependencies as additional packages to install. install.packages already does this for you, as it did in the construct of a repository code that I provided previously in the thread. You should be *only* doing
install.packages(<pkg in question>, repos = *) Then everything happens automatically via extremely well tested very
mature code.
I (still) don't understand why you'd need to pass install.packages the
vector of dependencies yourself, as that is counter to install.packages' core design.
Does that make sense? Best, ~G On Fri, Oct 28, 2022 at 12:18 PM Jan Gorecki <j.gorecki at wit.edu.pl>
wrote:
Gabriel,
I am trying to design generic solution that could be applied to
arbitrary package. Therefore I went with the latter solution you
proposed.
If we wouldn't have to exclude base packages, then its a 3 liner
file.copy("DESCRIPTION", file.path(tdir<-tempdir(), "PACKAGES"));
db<-available.packages(paste0("file://", tdir));
utils::install.packages(tools::package_dependencies("pkgname", db,
which="most")[[1L]])
As you noticed, we still have to filter out base packages. Otherwise
it won't be a robust utility that can be used in CI. Therefore we have
to add a call to tools:::.get_standard_package_names() which is an
internal function (as of now). Not only complicating the call but also
putting the functionality outside of safe use.
Considering above, don't you agree that the following one liner could
nicely address the problem? The problem that hundreds/thousands of
packages are now addressing in their CI scripts by using a third party
packages.
utils::install.packages(packages.dcf("DESCRIPTION", which="most"))
It is hard to me to understand why R members don't consider this basic
functionality to be part of base R. Possibly they just don't need it
themselves. Yet isn't this sufficient that hundreds/thousands of
packages does need this functionality?
Best regards,
Jan
On Mon, Oct 17, 2022 at 8:39 AM Jan Gorecki <j.gorecki at wit.edu.pl>
wrote:
Gabriel and Simon I completely agree with what you are saying. The thing is that obtaining recursive deps, all/most whatever, is
already well supported in core R. What is missing is just this single functionality I am requesting.
If you will look into the branch you can see there is mirror.packages
function meant to mirror a slice of CRAN. It is doing exactly what you described: package_dependencies; to obtain recursive deps, then download all, etc.
I would love to have this function provided by core R as well, but we
need to start somewhere.
There are other use cases as well. For example CI, where one wants to install all/most dependencies and
then run R CMD check. Then we don't worry about recursive deps are they will be resolved automatically.
I don't think it's reasonable to force users to use 3rd party
packages to handle such a common and simple use case. Otherwise one has to hard code deps in CI script. Not robust at all.
packages.dcf and repos.dcf makes all that way easier, and are solid
base for building customized orchestration like mirroring slice of CRAN.
Best regards Jan On Sun, Oct 16, 2022, 01:31 Simon Urbanek <
simon.urbanek at r-project.org> wrote:
Jan, I think using a single DCF as input is not very practical and would
not be useful in the context you describe (creating self contained repos) since they typically concern a list of packages, but essentially splitting out the part of install.packages() which determines which files will be pulled from where would be very useful as it would be trivial to use it to create repository (what we always do in corporate environments) instead of installing the packages. I suspect that install packages is already too complex so instead of adding a flag to install.packages one could move that functionality into a separate function - we all do that constantly for the sites we manage, so it would be certainly something worthwhile.
Cheers, Simon
On Oct 15, 2022, at 7:14 PM, Jan Gorecki <j.gorecki at wit.edu.pl>
wrote:
Hi Gabriel, It's very nice usage you provided here. Maybe instead of adding new function we could extend packages_depenedncies then? To accept
file path to
dsc file. What about repos.dcf? Maybe additional repositories could be an
attribute
attached to returned character vector. The use case is to, for a given package sources, obtain its
dependencies,
so one can use that for installing them/mirroring CRAN subset, or
whatever.
The later is especially important for a production environment
where one
wants to have fixed version of packages, and mirroring relevant
subset of
CRAN is the most simple, and IMO reliable, way to manage such
environment.
Regards Jan On Fri, Oct 14, 2022, 23:34 Gabriel Becker <gabembecker at gmail.com>
wrote:
Hi Jan and Jan, Can you explain a little more what exactly you want the
non-recursive,
non-version aware dependencies from an individual package for? Either way package_dependencies will do this for you* with a
little
"aggressive convincing". It wants output from available.packages,
but who
really cares what it wants? It's a function and we are people :)
library(tools)
db <- read.dcf("~/gabe/checkedout/rtables_clean/DESCRIPTION")
package_dependencies("rtables", db, which =
intersect(c("Depends",
"Suggests", "Imports", "LinkingTo"), colnames(db))) $rtables [1] "methods" "magrittr" "formatters" "dplyr" "tibble" [6] "tidyr" "testthat" "xml2" "knitr"
"rmarkdown"
[11] "flextable" "officer" "stats" "htmltools" "grid" The only gotcha that I see immediately is that "LinkingTo" isn't
always
there (whereas it is with real output from available.packages).
If you
know your package doesn't have that (or that it does) at call
time , this
becomes a one-liner:
package_dependencies("rtables", db =
read.dcf("~/gabe/checkedout/rtables_clean/DESCRIPTION"), which =
c("Depends", "Suggests", "Imports"))
$rtables
[1] "methods" "magrittr" "formatters" "dplyr" "tibble"
[6] "tidyr" "testthat" "xml2" "knitr"
"rmarkdown"
[11] "flextable" "officer" "stats" "htmltools" "grid" You can also trick it a slightly different way by giving it what
it
actually wants
tdir <- tempdir()
file.copy("~/gabe/checkedout/rtables_clean/DESCRIPTION",
file.path(tdir,
"PACKAGES")) [1] TRUE
avl <- available.packages(paste0("file://", tdir))
library(tools)
package_dependencies("rtables", avl)
$rtables [1] "methods" "magrittr" "formatters" "stats"
"htmltools"
[6] "grid"
package_dependencies("rtables", avl, which = "all")
$rtables [1] "methods" "magrittr" "formatters" "stats"
"htmltools"
[6] "grid" "dplyr" "tibble" "tidyr" "testthat" [11] "xml2" "knitr" "rmarkdown" "flextable" "officer" So the only real benefits I see that we'd be picking up here is
automatic
filtering by priority, and automatic extraction of the package
name from
the DESCRIPTION file. I'm not sure either of those warrant a new
exported
function that R-core has to maintain forever. Best, ~G * I haven't tested this across all OSes, but I dont' know of any
reason it
wouldn't work generally. On Fri, Oct 14, 2022 at 2:33 PM Jan Gorecki <j.gorecki at wit.edu.pl>
wrote:
Hello Jan, Thanks for confirming about many packages reinventing this
missing
functionality. packages.dcf was not meant handle versions. It just extracts
names of
dependencies... Yes, such a simple thing, yet missing in base R. Versions of packages can be controlled when setting up R pkgs
repo. This
is how I used to handle it. Making a CRAN subset mirror of fixed
version
pkgs. BTW. function for that is also included in mentioned branch. I
am just not
proposing it, to increase the chance of having at least this
simple,
missing, functionality merged. Best Jan On Fri, Oct 14, 2022, 15:14 Jan Net?k <netikja at gmail.com> wrote:
Hello Jan, I have seen many packages that implemented dependencies
"extraction" on
their own for internal purposes and today I was doing exactly
that for
mine. It's not a big deal using read.dcf on DESCRIPTION. It was
sufficient
for me, but I had to take care of some \n chars (the overall
returned
value
has some rough edges, in my opinion). However, the function
from the
branch
seems to not care about version requirements, which are crucial
for me.
Maybe that is something to reconsider before merging. Best, Jan p? 14. 10. 2022 v 2:27 odes?latel Jan Gorecki <
j.gorecki at wit.edu.pl>
napsal:
Dear R devs, I would like to raise a request for a simple helper function. Utility function to extract package dependencies from
DESCRIPTION file.
I do think that tools package is better place, for such a
fundamental
functionality, than community packages. tools pkg seems perfect fit (having already great function write_PACKAGES). Functionality I am asking for is already in R svn repository
since
2016,
in a branch tools4pkgs. Function is called 'packages.dcf'. Another one 'repos.dcf' would be a good functional
complementary to it.
Those two simple helper functions really makes it easier for
organizations
to glue together usage of their own R packages repos and CRAN
repo in a
smooth way. That could possibly help to offload CRAN from new
submissions.
gh mirror link for easy preview:
Regards
Jan Gorecki
[[alternative HTML version deleted]]
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
[[alternative HTML version deleted]]
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
[[alternative HTML version deleted]]
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Thank you Gabriel,
Just for future readers. Below is a base R way to address this common
problem, as instructed by you (+stopifnot to suppress print).
Rscript -e 'stopifnot(file.copy("DESCRIPTION",
file.path(tdir<-tempdir(), "PACKAGES")));
db<-available.packages(paste0("file://", tdir));
install.packages(setdiff(tools::package_dependencies(read.dcf("DESCRIPTION",
fields="Package")[[1L]], db, which="most")[[1L]],
installed.packages(priority="high")[,"Package"]))'
3 liner, 310 chars long command, far from ideal, but does work.
Best,
Jan
On Fri, Oct 28, 2022 at 10:42 PM Gabriel Becker <gabembecker at gmail.com> wrote:
Hi Jan, On Fri, Oct 28, 2022 at 1:57 PM Jan Gorecki <j.gorecki at wit.edu.pl> wrote:
Gabriel, It is the most basic CI use case. One wants to install only dependencies only of the package, and run R CMD check on the package.
Really what you're looking for though, is to install all the dependencies which aren't present right? Excluding base packages is just a particular way to do that under certain assumptions about the CI environment.
So
needed_pkgs <- setdiff(package_dependencies(...), installed.packages()[,"Package"])
install.packages(needed_pkgs, repos = fancyrepos)
will do what you want without installing the package itself, if that is important. This will filter out base and recommended packages (which will be already installed in your CI container, since R is).
Now this does not take into account versioned dependencies, so it's not actually fully correct (whereas installing the package is), but it gets you where you're trying to go. And in a clean CI container without cached package installation for the deps, its equivalent.
Also, as an aside, if you need to get the base packages, you can do
installed.packages(priority="base")[,"Package"]
base compiler datasets graphics grDevices grid
"base" "compiler" "datasets" "graphics" "grDevices" "grid"
methods parallel splines stats stats4 tcltk
"methods" "parallel" "splines" "stats" "stats4" "tcltk"
tools utils
"tools" "utils"
(to get base and recommended packages use 'high' instead of 'base')
No need to be reaching down into unexported functions. So if you *really* only want to exclude base functions (which likely will give you some protection from versioned dep issues), you can change the code above to
needed_pkgs <- setdiff(package_dependencies(...), installed.packages(priority = "high")[,"Package"])
install.packages(needed_pkgs, repos = fancyrepos)
Best,
~G
On Fri, Oct 28, 2022 at 8:42 PM Gabriel Becker <gabembecker at gmail.com> wrote:
Hi Jan, The reason, I suspect without speaking for R-core, is that by design you should not be specifying package dependencies as additional packages to install. install.packages already does this for you, as it did in the construct of a repository code that I provided previously in the thread. You should be *only* doing install.packages(<pkg in question>, repos = *) Then everything happens automatically via extremely well tested very mature code. I (still) don't understand why you'd need to pass install.packages the vector of dependencies yourself, as that is counter to install.packages' core design. Does that make sense? Best, ~G On Fri, Oct 28, 2022 at 12:18 PM Jan Gorecki <j.gorecki at wit.edu.pl> wrote:
Gabriel,
I am trying to design generic solution that could be applied to
arbitrary package. Therefore I went with the latter solution you
proposed.
If we wouldn't have to exclude base packages, then its a 3 liner
file.copy("DESCRIPTION", file.path(tdir<-tempdir(), "PACKAGES"));
db<-available.packages(paste0("file://", tdir));
utils::install.packages(tools::package_dependencies("pkgname", db,
which="most")[[1L]])
As you noticed, we still have to filter out base packages. Otherwise
it won't be a robust utility that can be used in CI. Therefore we have
to add a call to tools:::.get_standard_package_names() which is an
internal function (as of now). Not only complicating the call but also
putting the functionality outside of safe use.
Considering above, don't you agree that the following one liner could
nicely address the problem? The problem that hundreds/thousands of
packages are now addressing in their CI scripts by using a third party
packages.
utils::install.packages(packages.dcf("DESCRIPTION", which="most"))
It is hard to me to understand why R members don't consider this basic
functionality to be part of base R. Possibly they just don't need it
themselves. Yet isn't this sufficient that hundreds/thousands of
packages does need this functionality?
Best regards,
Jan
On Mon, Oct 17, 2022 at 8:39 AM Jan Gorecki <j.gorecki at wit.edu.pl> wrote:
Gabriel and Simon I completely agree with what you are saying. The thing is that obtaining recursive deps, all/most whatever, is already well supported in core R. What is missing is just this single functionality I am requesting. If you will look into the branch you can see there is mirror.packages function meant to mirror a slice of CRAN. It is doing exactly what you described: package_dependencies; to obtain recursive deps, then download all, etc. I would love to have this function provided by core R as well, but we need to start somewhere. There are other use cases as well. For example CI, where one wants to install all/most dependencies and then run R CMD check. Then we don't worry about recursive deps are they will be resolved automatically. I don't think it's reasonable to force users to use 3rd party packages to handle such a common and simple use case. Otherwise one has to hard code deps in CI script. Not robust at all. packages.dcf and repos.dcf makes all that way easier, and are solid base for building customized orchestration like mirroring slice of CRAN. Best regards Jan On Sun, Oct 16, 2022, 01:31 Simon Urbanek <simon.urbanek at r-project.org> wrote:
Jan, I think using a single DCF as input is not very practical and would not be useful in the context you describe (creating self contained repos) since they typically concern a list of packages, but essentially splitting out the part of install.packages() which determines which files will be pulled from where would be very useful as it would be trivial to use it to create repository (what we always do in corporate environments) instead of installing the packages. I suspect that install packages is already too complex so instead of adding a flag to install.packages one could move that functionality into a separate function - we all do that constantly for the sites we manage, so it would be certainly something worthwhile. Cheers, Simon
On Oct 15, 2022, at 7:14 PM, Jan Gorecki <j.gorecki at wit.edu.pl> wrote: Hi Gabriel, It's very nice usage you provided here. Maybe instead of adding new function we could extend packages_depenedncies then? To accept file path to dsc file. What about repos.dcf? Maybe additional repositories could be an attribute attached to returned character vector. The use case is to, for a given package sources, obtain its dependencies, so one can use that for installing them/mirroring CRAN subset, or whatever. The later is especially important for a production environment where one wants to have fixed version of packages, and mirroring relevant subset of CRAN is the most simple, and IMO reliable, way to manage such environment. Regards Jan On Fri, Oct 14, 2022, 23:34 Gabriel Becker <gabembecker at gmail.com> wrote:
Hi Jan and Jan, Can you explain a little more what exactly you want the non-recursive, non-version aware dependencies from an individual package for? Either way package_dependencies will do this for you* with a little "aggressive convincing". It wants output from available.packages, but who really cares what it wants? It's a function and we are people :)
library(tools)
db <- read.dcf("~/gabe/checkedout/rtables_clean/DESCRIPTION")
package_dependencies("rtables", db, which = intersect(c("Depends",
"Suggests", "Imports", "LinkingTo"), colnames(db)))
$rtables
[1] "methods" "magrittr" "formatters" "dplyr" "tibble"
[6] "tidyr" "testthat" "xml2" "knitr" "rmarkdown"
[11] "flextable" "officer" "stats" "htmltools" "grid"
The only gotcha that I see immediately is that "LinkingTo" isn't always
there (whereas it is with real output from available.packages). If you
know your package doesn't have that (or that it does) at call time , this
becomes a one-liner:
package_dependencies("rtables", db =
read.dcf("~/gabe/checkedout/rtables_clean/DESCRIPTION"), which =
c("Depends", "Suggests", "Imports"))
$rtables
[1] "methods" "magrittr" "formatters" "dplyr" "tibble"
[6] "tidyr" "testthat" "xml2" "knitr" "rmarkdown"
[11] "flextable" "officer" "stats" "htmltools" "grid"
You can also trick it a slightly different way by giving it what it
actually wants
tdir <- tempdir()
file.copy("~/gabe/checkedout/rtables_clean/DESCRIPTION", file.path(tdir,
"PACKAGES")) [1] TRUE
avl <- available.packages(paste0("file://", tdir))
library(tools)
package_dependencies("rtables", avl)
$rtables [1] "methods" "magrittr" "formatters" "stats" "htmltools" [6] "grid"
package_dependencies("rtables", avl, which = "all")
$rtables [1] "methods" "magrittr" "formatters" "stats" "htmltools" [6] "grid" "dplyr" "tibble" "tidyr" "testthat" [11] "xml2" "knitr" "rmarkdown" "flextable" "officer" So the only real benefits I see that we'd be picking up here is automatic filtering by priority, and automatic extraction of the package name from the DESCRIPTION file. I'm not sure either of those warrant a new exported function that R-core has to maintain forever. Best, ~G * I haven't tested this across all OSes, but I dont' know of any reason it wouldn't work generally. On Fri, Oct 14, 2022 at 2:33 PM Jan Gorecki <j.gorecki at wit.edu.pl> wrote:
Hello Jan, Thanks for confirming about many packages reinventing this missing functionality. packages.dcf was not meant handle versions. It just extracts names of dependencies... Yes, such a simple thing, yet missing in base R. Versions of packages can be controlled when setting up R pkgs repo. This is how I used to handle it. Making a CRAN subset mirror of fixed version pkgs. BTW. function for that is also included in mentioned branch. I am just not proposing it, to increase the chance of having at least this simple, missing, functionality merged. Best Jan On Fri, Oct 14, 2022, 15:14 Jan Net?k <netikja at gmail.com> wrote:
Hello Jan, I have seen many packages that implemented dependencies "extraction" on their own for internal purposes and today I was doing exactly that for mine. It's not a big deal using read.dcf on DESCRIPTION. It was
sufficient
for me, but I had to take care of some \n chars (the overall returned
value
has some rough edges, in my opinion). However, the function from the
branch
seems to not care about version requirements, which are crucial for me. Maybe that is something to reconsider before merging. Best, Jan p? 14. 10. 2022 v 2:27 odes?latel Jan Gorecki <j.gorecki at wit.edu.pl> napsal:
Dear R devs, I would like to raise a request for a simple helper function. Utility function to extract package dependencies from DESCRIPTION file. I do think that tools package is better place, for such a fundamental functionality, than community packages. tools pkg seems perfect fit (having already great function write_PACKAGES). Functionality I am asking for is already in R svn repository since
2016,
in a branch tools4pkgs. Function is called 'packages.dcf'. Another one 'repos.dcf' would be a good functional complementary to it. Those two simple helper functions really makes it easier for
organizations
to glue together usage of their own R packages repos and CRAN repo in a smooth way. That could possibly help to offload CRAN from new
submissions.
gh mirror link for easy preview:
Regards
Jan Gorecki
[[alternative HTML version deleted]]
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
[[alternative HTML version deleted]]
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
[[alternative HTML version deleted]]
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel