Martin, Vince, Sean,
thank you very much for your comments and suggestions, i've looked at
the package 'itdepends' from Jim Hester, this was a great suggestion. i
actually found a talk he gave about it on rstudioconf2019, here:
https://resources.rstudio.com/rstudio-conf-2019/it-depends-a-dialog-about-dependencies
i recommend watching it to anyone interested in this thread, i think
pretty much tackles the most important issues we're concerned as
developers, regarding dependencies.
ironically, the package 'itdepends' doesn't seem to be actively
developed: it's not part of CRAN, the GitHub repo hasn't been updated in
the last 5 months, it has 10 open issues for 5 closed ones and i've
experienced that some functions break in the current R-devel.
i also didn't know about 'BiocPkgTools' and this seems to be the right
home for adding the kind of functionality we're talking about, although
i would think the same for 'itdepends' if it would be pushed to CRAN at
some point.
i've invested some time to develop what it constitutes at the moment my
own needs on this subject. in case this is useful to anyone i've made a
GitHub gist available here:
https://gist.github.com/rcastelo/7429d05178ddb57a38bd42093c2ddfe2
i haven't attempted to integrate this into 'BiocPkgTools' and do a pull
request because of two reasons:
1. if i try to fetch the dependencies from CRAN, as well as from BioC
(which is the only default), i get an error:
library(BiocPkgTools)
df <- buildPkgDependencyDataFrame(repo=c("BioCsoft", "CRAN"))
Error in url(viewsFileUrl) : invalid 'description' argument
2. because some of the calls break 'itdepends' in R-devel, this would
also break 'BiocPkgTools' in R-devel. i'm also not sure how feasible it
is for a BioC package to have a package dependency outside CRAN and BioC.
my initial motivation for all this was that the installation of
'GenomicScores' was breaking in one of our servers because of
compilation problems with the package 'Matrix'. this was surprising to
me because i wasn't expecting to have that dependency. after the first
exchange of messages in this thread, using the code we wrote, i
identified that only a few lines in the source of 'GenomicScores' were
leading to that dependency upstream. i could replace them and get rid of
that dependency and actually other ones.
i've tried to provide a first attempt for a general approach to this
situation. first we should source the gist:
devtools::source_gist("rcastelo/depburden.R")
then build a database of dependencies information:
repos <- BiocManager::repositories()[c("BioCsoft", "CRAN")]
db <- utils::available.packages(repos=repos)
and now the important part consists of the following three steps:
1. identify the burden of dependencies of a package, e.g., "GenomicScores"
pkgDepMetrics("GenomicScores", db)
ImportedBy Exported Usage DepOverlap
Biobase 1 128 0.781250 0.0250
BSgenome 1 93 1.075269 0.3625
XML 2 175 1.142857 0.0125
IRanges 4 254 1.574803 0.0375
BiocGenerics 5 139 3.597122 0.0125
GenomicRanges 4 104 3.846154 0.1125
S4Vectors 11 262 4.198473 0.0250
GenomeInfoDb 5 53 9.433962 0.0750
AnnotationHub 4 33 12.121212 0.6875
Biostrings NA 240 NA 0.0750
following Jim's recommendations on his talk, concretely those in minute
16, this function reports the number of function calls to a dependency
and the number of exported functions by that dependency. the column
'Usage' is the percentage of those imported calls to the exposed
functionality by the dependency. for instance, if i want to get rid of
'AnnotationHub' i'd have to implement in my package about the 12% of the
functionality exported by 'AnnotationHub'.
the column 'DepOverlap' shows the overlap between the dependency graph
of the analyzed package and the dependency graph of the dependency in
that row. this is calculated as a Jaccard index (intersection of
vertices divided by the union) where 0 would correspond to disjoint
graphs and 1 to identical ones.
from these numbers i can see that, for instance, i'm importing just one
function call from 'BSgenome' (about 1% of its functionality), while the
dependency burden of 'BSGenome' overlaps more than 1/3 of the total
burden of the package. this is to me a good candidate to explore in the
following two steps.
2.let's say we want to investigate what function calls are responsible
for the dependency on "BSgenome"
funCalls2Dep("GenomicScores", "BSgenome", db)
# A tibble: 1 x 3
# Groups: pkg [1]
pkg fun n
<chr> <chr> <int>
1 BSgenome referenceGenome 4
so i'm using a function or method called "referenceGenome" imported from
"BSgenome"
3. we want now to see what lines in our code contain those function
calls (assuming we're in the source path of the package "GenomicScores"):
lines <- funCalls2Dep("GenomicScores", "BSgenome", db, ".", "R")
head(lines, 2)
[[1]]
R/makeGScoresPackage.R:60:68: warning: BSgenome::referenceGenome
organism(gsco),
providerVersion(referenceGenome(gsco))),
^~~~~~~~~~~~~~~
[[2]]
R/makeGScoresPackage.R:69:49: warning: BSgenome::referenceGenome
GENOMEVERSION=providerVersion(referenceGenome(gsco)),
^~~~~~~~~~~~~~~
here i'm using the release version of R because otherwise, as i said
before, some of the function calls to the 'itdepends' package break.
i'd be happy to pull-request this code, with the necessary adaptations,
wherever the community feels is more appropriate, but i'd say that the
problem with 'itdepends' and R-devel should be fixed first, and then we
can decide if this is something we want to incorporate into an API and
from what package.
cheers,
robert.
On 2/9/20 5:01 PM, Sean Davis wrote:
There are some good ideas here that would provide enhancement to
BiocPkgTools. I don't have the bandwidth to incorporate right now, but
filing issues or a pull request with a skeleton would be helpful to keep
track.
Sean
On Sun, Feb 9, 2020 at 7:31 AM Vincent Carey <stvjc at channing.harvard.edu
wrote:
On Sat, Feb 8, 2020 at 12:02 PM Martin Morgan <mtmorgan.bioc at gmail.com>
wrote:
I find it quite interesting to identify formal strategies for removing
dependencies, but also a little outside my domain of expertise. This
It would be nice to collect the ideas in this thread into some
recommendations. The themes I am thinking of
are "how developers can make their packages robust to loss of external
packages" and "how can the
Bioc ecosystem best deal with departures of packages from itself and
CRAN?" A good and well-adopted
solution to the first one makes the second one moot.
Two CRAN-related events I know of that required some effort are
loss of ashr and (recently)
archiving of Seurat.
library(tools)
library(dplyr)
## non-base packages the user requires for GenomicScores
deps <- package_dependencies("GenomicScores", db, recursive=TRUE)[[1]]
deps <- intersect(deps, rownames(db))
## only need the 'universe' of GenomicScores dependencies
db1 <- db[c("GenomicScores", deps),]
## sub-graph of packages between each dependency and GenomicScores
revdeps <- package_dependencies(deps, db1, recursive = TRUE, reverse =
TRUE)
tibble(
package = names(olap),
n_remove = lengths(revdeps),
) %>%
arrange(n_remove)
produces a tibble
# A tibble: 106 x 2
package n_remove
<chr> <int>
1 BSgenome 1
2 AnnotationHub 1
3 shinyjs 1
4 DT 1
5 shinycustomloader 1
6 data.table 1
7 shinythemes 1
8 rtracklayer 2
9 BiocFileCache 2
10 BiocManager 2
# ? with 96 more rows
shows me, via n_remove, that I can remove the dependency on
by removing the dependency on just one package (AnnotationHub!), but to
remove BiocFileCache I'd also have to remove another package
(AnnotationHub, I'd guess). So this provides some measure of the ease
which a package can be removed.
I'd like a 'benefit' column, too -- if I were to remove AnnotationHub,
many additional packages would I also be able to remove, because they
present only to satisfy the dependency on AnnotationHub? More
perhaps there is a dependency of AnnotationHub that is only used by
AnnotationHub and BSgenome. So removing AnnotationHub as a dependency
make it easier to remove BSgenome, etc. I guess this is a graph
optimization problem.
Probably also worth mentioning the itdepends package (
https://github.com/r-lib/itdepends), which I think tries primarily to
determine the relationship between package dependencies and lines of
which seems like complementary information.
Martin
?On 2/6/20, 12:29 PM, "Robert Castelo" <robert.castelo at upf.edu> wrote:
true, i was just searching for the shortest path, we can search
all
simple (i.e., without repeating "vertices") paths and there are
five routes from "GenomicScores" to "Matrix"
igraph::all_simple_paths(igraph::igraph.from.graphNEL(g),
from="GenomicScores", to="Matrix", mode="out")
[[1]]
+ 7/117 vertices, named, from 04133ec:
[1] GenomicScores BSgenome rtracklayer
[4] GenomicAlignments SummarizedExperiment DelayedArray
[7] Matrix
[[2]]
+ 6/117 vertices, named, from 04133ec:
[1] GenomicScores BSgenome rtracklayer
[4] GenomicAlignments SummarizedExperiment Matrix
[[3]]
+ 6/117 vertices, named, from 04133ec:
[1] GenomicScores DT crosstalk ggplot2 mgcv
[6] Matrix
[[4]]
+ 6/117 vertices, named, from 04133ec:
[1] GenomicScores rtracklayer GenomicAlignments
[4] SummarizedExperiment DelayedArray Matrix
[[5]]
+ 5/117 vertices, named, from 04133ec:
[1] GenomicScores rtracklayer GenomicAlignments
[4] SummarizedExperiment Matrix
this is interesting, because it means that if i wanted to get rid
the
"Matrix" dependence i'd need to get rid not only of the
dependence but also of "BSgenome" and "DT".
robert.
On 2/6/20 5:41 PM, Martin Morgan wrote:
> Excellent! I think there are other, independent, paths between
>
> RBGL::sp.between(g, start="DT", finish="Matrix",
detail=TRUE)[[1]]$path_detail
> [1] "DT" "crosstalk" "ggplot2" "mgcv" "Matrix"
>
> ??
>
> Martin
>
> ?On 2/6/20, 10:47 AM, "Robert Castelo" <robert.castelo at upf.edu>
>
> hi Martin,
>
> thanks for hint!! i wasn't aware of
'tools::package_dependencies()',
> adding a bit of graph sorcery i get the result i was
>
> repos <- BiocManager::repositories()[c(1,5)]
> repos
> BioCsoft
> "https://bioconductor.org/packages/3.11/bioc"
> CRAN
> "https://cran.rstudio.com"
>
> db <- available.packages(repos=repos)
>
> deps <- tools::package_dependencies("GenomicScores", db,
> recursive=TRUE)[[1]]
>
> deps <- tools::package_dependencies(c("GenomicScores",
>
> g <- graph::graphNEL(nodes=names(deps), edgeL=deps,
>
> RBGL::sp.between(g, start="GenomicScores", finish="Matrix",
> detail=TRUE)[[1]]$path_detail
> [1] "GenomicScores" "rtracklayer"
> [4] "SummarizedExperiment" "Matrix"
>
> so, it was the rtracklayer dependency that leads to Matrix
> GenomeAlignments and SummarizedExperiment.
>
> maybe the BioC package 'pkgDepTools' should be deprecated
> functionality is part of 'tools' and it does not even work
> correct as 'tools'.
>
> cheers,
>
> robert.
>
>
> On 2/6/20 2:51 PM, Martin Morgan wrote:
> > The first thing is to get the correct repositories
> >
> > repos = BiocManager::repositories()
> >
> > (maybe trim the experiment and annotation repos from
also tried pkgDepTools::makeDepGraph() but it took so long that I moved
on... it has an option 'keep.builtin' which might include Matrix.
BiocPkgTools::buildPkgDependencyDataFrame() &
friends, but this seems to build dependencies within a single
> >
> > The building block for a solution is
`tools::package_dependencies()`, and I can confirm that "Matrix" _is_ a
dependency
> >
> > db = available.packages(repos =
BiocManager::repositories())
tools::package_dependencies("GenomicScores",
> > "Matrix" %in% revdeps[[1]]
> > ## [1] TRUE
> >
> > so I'll leave the clever recursive or graph-based
up to you, to report back to the mailing list?
> >
> > For what it's worth I think the last time this came up
Maechler pointed to a function in base R (probably the tools package)
> >
> > Martin Morgan
> >
> > ?On 2/6/20, 6:40 AM, "Bioc-devel on behalf of Robert
<bioc-devel-bounces at r-project.org on behalf of robert.castelo at upf.edu>
wrote:
> >
> > hi,
> >
> > when i load the package 'GenomicScores' in a clean
> > the 'sessionInfo()' that the package 'Matrix' is
> > via a namespace (and not attached)".
> >
> > i'd like to know what is the dependency that
'GenomicsScores' has that
> > ends up requiring the package 'Matrix'.
> >
> > i've tried using the package 'pkgDepTools' without
> > dependency graph does not list any path from
'GenomicScores' to 'Matrix'.
> >
> > i've been manually browsing the Bioc website and,
> > something, the only association with 'Matrix' i
> > 'S4Vectors' and 'GenomicRanges', which are required
> > list 'Matrix' in the 'Suggests' field, but my
> > those packages are not required and should not be
> >
> > so, is there any way in which i can figure out what
> > 'GenomicScores' dependencies leads to loading the
> >
> > here are the depends, import and suggests fields
> >
> > Depends: R (>= 3.5), S4Vectors (>= 0.7.21),
> > BiocGenerics (>= 0.13.8)
> > Imports: utils, XML, Biobase, IRanges (>= 2.3.23),
> > BSgenome, GenomeInfoDb, AnnotationHub,
> > DT, shinycustomloader, rtracklayer,
> > Suggests: BiocStyle, knitr, rmarkdown,
BSgenome.Hsapiens.UCSC.hg19,
> > phastCons100way.UCSC.hg19,
MafDb.1Kgenomes.phase1.hs37d5,
> > SNPlocs.Hsapiens.dbSNP144.GRCh37,
> > TxDb.Hsapiens.UCSC.hg19.knownGene, gwascat,
> >
> > and here a session information in a fresh R-devel
> > the package 'GenomicScores':
> >
> > R Under development (unstable) (2020-01-29 r77745)
> > Platform: x86_64-pc-linux-gnu (64-bit)
> > Running under: CentOS Linux 7 (Core)
> >
> > Matrix products: default
> > BLAS: /opt/R/R-devel/lib64/R/lib/libRblas.so
> > LAPACK: /opt/R/R-devel/lib64/R/lib/libRlapack.so
> >
> > locale:
> > [1] LC_CTYPE=en_US.UTF8 LC_NUMERIC=C
> > [3] LC_TIME=en_US.UTF8
> > [5] LC_MONETARY=en_US.UTF8
> > [7] LC_PAPER=en_US.UTF8 LC_NAME=C
> > [9] LC_ADDRESS=C LC_TELEPHONE=C
> > [11] LC_MEASUREMENT=en_US.UTF8 LC_IDENTIFICATION=C
> >
> > attached base packages:
> > [1] parallel stats4 stats graphics
> > [8] methods base
> >
> > other attached packages:
> > [1] GenomicScores_1.11.4 GenomicRanges_1.39.2
> > [4] IRanges_2.21.3 S4Vectors_0.25.12
> > [7] colorout_1.2-2
> >
> > loaded via a namespace (and not attached):
> > [1] Rcpp_1.0.3 lattice_0.20-38
> > [3] shinycustomloader_0.9.0 Rsamtools_2.3.3
> > [5] Biostrings_2.55.4 assertthat_0.2.1
> > [7] digest_0.6.23 mime_0.9
> > [9] BiocFileCache_1.11.4 R6_2.4.1
> > [11] RSQLite_2.2.0 httr_1.4.1
> > [13] pillar_1.4.3 zlibbioc_1.33.1
> > [15] rlang_0.4.4 curl_4.3
> > [17] data.table_1.12.8 blob_1.2.1
> > [19] DT_0.12 Matrix_1.2-18
> > [21] shinythemes_1.1.2 shinyjs_1.1
> > [23] BiocParallel_1.21.2
> > [25] htmlwidgets_1.5.1 RCurl_1.98-1.1
> > [27] bit_1.1-15.1 shiny_1.4.0
> > [29] DelayedArray_0.13.3 compiler_4.0.0
> > [31] httpuv_1.5.2
> > [33] pkgconfig_2.0.3 htmltools_0.4.0
> > [35] tidyselect_1.0.0
SummarizedExperiment_1.17.1
> > [39] interactiveDisplayBase_1.25.0
> > [41] XML_3.99-0.3 crayon_1.3.4
> > [43] dplyr_0.8.4 dbplyr_1.4.2
> > [45] later_1.0.0
> > [47] bitops_1.0-6 rappdirs_0.3.1
> > [49] grid_4.0.0 xtable_1.8-4
> > [51] DBI_1.1.0 magrittr_1.5
> > [53] XVector_0.27.0 promises_1.1.0
> > [55] vctrs_0.2.2 tools_4.0.0
> > [57] bit64_0.9-7 BSgenome_1.55.3
> > [59] Biobase_2.47.2 glue_1.3.1
> > [61] purrr_0.3.3
> > [63] fastmap_1.0.1 yaml_2.2.1
> > [65] AnnotationDbi_1.49.1
>
> --
> Robert Castelo, PhD
> Associate Professor
> Dept. of Experimental and Health Sciences
> Universitat Pompeu Fabra (UPF)
> Barcelona Biomedical Research Park (PRBB)
> Dr Aiguader 88
> E-08003 Barcelona, Spain
> telf: +34.933.160.514
> fax: +34.933.160.550
>
>
--
Robert Castelo, PhD
Associate Professor
Dept. of Experimental and Health Sciences
Universitat Pompeu Fabra (UPF)
Barcelona Biomedical Research Park (PRBB)
Dr Aiguader 88
E-08003 Barcelona, Spain
telf: +34.933.160.514
fax: +34.933.160.550