R 4.0.0 rebuild status

On Monday, 29 June 2020 13.46.02 WEST I?aki Ucar wrote:

But the mass rebuild process is very different, because releng doesn't
follow any particular order (they don't need to, because nothing
really changed).

The question is whether there is any tool to resolve dependencies and
generate a list of builds, as I do with the CRAN database, for an
arbitrary list of RPMs. That would be perfect. In theory, this should
be possible with dnf, but when I tried, I failed (or my script would
theoretically work, but it would take ages to complete). I also took a
look at releng's repos on Pagure, but I didn't find any tool to do
this either.

We should bring this to devel, maybe ask Miro and other Python folks
who may have The Method(TM), as they are rebuilding stuff on a daily
basis.

I think that I meant the way that the python packages are rebuilt for each new
python version. I should have said Python mass rebuild.

I took your work as the basis, together with Tom's help and I used dnf to get
a list of R packages (assuming that their name starts with R- ). The advantage
of this approach is that it works for CRAN and Bioconductor.

$ dnf list available --disablerepo=\* --enablerepo=rawhide-source R-* | tail -
n +3 | awk -e '{print $1};' | sed -e 's/\.src$//' > r-packages.txt

That gave the list of the (source) packages to rebuild.

In order to get the list of dependencies I used a similar strategy:
$ (for p in $(cat r-packages.txt); do echo -n $p, ; dnf -q repoquery --
disablerepo=\* --enablerepo=rawhide-source --requires $p | sed ':a;N;$!ba;s/
\n/,/g'; done) > r-packages-dependencies.csv

The resulting file is a csv file with the first column as the package and the
remaining columns as the dependencies.

Now the idea was to process this file using a python script to create a direct
graph of dependencies (using networkx). In the process non R packages are
removed and some dependencies are cleaned (e.g. -devel packages direct to the
source package).

The advantage of this is that it allows to identify cycles.

My only surprise in this process was to find that several packages require R-
rpm-macros.

That causes some dependency cycles because R-rpm-macros also requires
R-rprintf,R-devel,R-knitr,R-stringi,R-testthat.

Since R-rpm-macros is used for Fedora and epel 8 I suggest to remove the
dependency. Do you agree?

On Monday, 29 June 2020 13.46.02 WEST I?aki Ucar wrote:

But the mass rebuild process is very different, because releng doesn't
follow any particular order (they don't need to, because nothing
really changed).

The question is whether there is any tool to resolve dependencies and
generate a list of builds, as I do with the CRAN database, for an
arbitrary list of RPMs. That would be perfect. In theory, this should
be possible with dnf, but when I tried, I failed (or my script would
theoretically work, but it would take ages to complete). I also took a
look at releng's repos on Pagure, but I didn't find any tool to do
this either.

We should bring this to devel, maybe ask Miro and other Python folks
who may have The Method(TM), as they are rebuilding stuff on a daily
basis.

I think that I meant the way that the python packages are rebuilt for each new
python version. I should have said Python mass rebuild.

I took your work as the basis, together with Tom's help and I used dnf to get
a list of R packages (assuming that their name starts with R- ). The advantage
of this approach is that it works for CRAN and Bioconductor.

$ dnf list available --disablerepo=\* --enablerepo=rawhide-source R-* | tail -
n +3 | awk -e '{print $1};' | sed -e 's/\.src$//' > r-packages.txt

That gave the list of the (source) packages to rebuild.

In order to get the list of dependencies I used a similar strategy:
$ (for p in $(cat r-packages.txt); do echo -n $p, ; dnf -q repoquery --
disablerepo=\* --enablerepo=rawhide-source --requires $p | sed ':a;N;$!ba;s/
\n/,/g'; done) > r-packages-dependencies.csv

The resulting file is a csv file with the first column as the package and the
remaining columns as the dependencies.

Now the idea was to process this file using a python script to create a direct
graph of dependencies (using networkx). In the process non R packages are
removed and some dependencies are cleaned (e.g. -devel packages direct to the
source package).


The advantage of this is that it allows to identify cycles.

My only surprise in this process was to find that several packages require R-
rpm-macros.

That causes some dependency cycles because R-rpm-macros also requires
R-rprintf,R-devel,R-knitr,R-stringi,R-testthat.

Since R-rpm-macros is used for Fedora and epel 8 I suggest to remove the
dependency. Do you agree?


--
Jos? Ab?lio

Your process must have an error, because R-rpm-macros only Requires
R-core. It also BuildRequires nothing.

R-devel depends on R-rpm-macros, and this is intended, in order to
pull in automated dependency generators. If anything else directly
requires R-rpm-macros, that is an error, and these are all in packages
I don't own.

Nice! What if we create a group "R" on Pagure and a repo
"fedora-scripts" or something like that?

On Friday, 3 July 2020 18.36.17 WEST I?aki Ucar wrote:

Nice! What if we create a group "R" on Pagure and a repo
"fedora-scripts" or something like that?

I would like to improve the scripts but FWIW here it comes a rough version of
the script I used.

The python script loads the csv file and constructs a directed graph and it
starts to remove the leave packages (pruning or trimming if you take a
gardening analogy).

The csv files has the dependency of the packages. It is always possible to go
to the packages that remain after the first stage and using a bootstrap mode
to remove most of the dependencies and then to run the code again.

This procedure was enough to process all the packages.

On Friday, 3 July 2020 18.36.17 WEST I?aki Ucar wrote:

Nice! What if we create a group "R" on Pagure and a repo
"fedora-scripts" or something like that?

I would like to improve the scripts but FWIW here it comes a rough version of
the script I used.

The python script loads the csv file and constructs a directed graph and it
starts to remove the leave packages (pruning or trimming if you take a
gardening analogy).

The csv files has the dependency of the packages. It is always possible to go
to the packages that remain after the first stage and using a bootstrap mode
to remove most of the dependencies and then to run the code again.

This procedure was enough to process all the packages.
--
Jos? Ab?lio

On Tue, 21 Jul 2020 at 11:05, Jos? Ab?lio Matos <jamatos at fc.up.pt> wrote:

On Friday, 3 July 2020 18.36.17 WEST I?aki Ucar wrote:

Nice! What if we create a group "R" on Pagure and a repo
"fedora-scripts" or something like that?

I would like to improve the scripts but FWIW here it comes a rough version of
the script I used.

The python script loads the csv file and constructs a directed graph and it
starts to remove the leave packages (pruning or trimming if you take a
gardening analogy).

I'm not exactly fluent in Haskell, but Jens Petersen has a tool called
fbrnch [1], which is able to calculate the build order, and then
submit them, and even handles override for non-Rawhide builds. It does
need you to handle some of the bootstrap conditions yourself, but with
a defined workflow, it might be possible to request that it handle
that too.