Skip to content

R 4.0.0 rebuild status

32 messages · Iñaki Ucar, Tom Callaway, José Abílio Matos +1 more

Messages 1–25 of 32

#
Over the last several days, I've been working hard to get all of the Fedora
R packages rebuilt against R 4.0 in rawhide (in the F33-R-4 side tag). With
the exception of R-biomaRt, R-BSgenome, R-GenomicAlignments, and
R-rtracklayer, I believe everything is built and updated to the latest
versions. And of those packages, they're all ready to go when Fedora
infrastructure is working reliably again (the great datacenter migration
has started and I can no longer git push). I'll also push R 4.0.1 into the
tag when that's possible.

There are two new packages that I created that are needed for R-biomaRt to
be updated:

R-AnnotationDbi
https://bugzilla.redhat.com/show_bug.cgi?id=1845360

R-BiocFileCache
https://bugzilla.redhat.com/show_bug.cgi?id=1845362

They are relatively simple noarch packages and should be quick reviews.
Help in getting these reviewed quickly would be appreciated.

Given the huge amount of builds (and rebuilds) in this process, I am
strongly disinclined to attempt this work for Fedora 32 (the idea of
hundreds of bodhi overrides does not fill me with joy), but I would not
prevent someone else who wished to try to do so.

Thanks,
Tom

P.S. I noticed that R-pls and R-statmod were orphaned (and as a result,
they did not get rebuilt for R 4). Nothing seems to depend on them in
Fedora, so this seems fine, but if you were using those packages, you may
wish to revive them.
#
On Tue, 9 Jun 2020 at 04:42, Tom Callaway <tcallawa at redhat.com> wrote:
Much appreciated.
I took them. In progress.
Note that this is no longer needed. It is possible to use a side tag
for F32 too, which is much easier and more appropriate for a massive
update like this.
14 days later
#
On Tue, 9 Jun 2020 at 10:21, I?aki Ucar <iucar at fedoraproject.org> wrote:
Any plan on doing this in a side tag for F32? I would happily
volunteer, but I'm not a provenpackager.
#
At this point, I simply don't have the time.

Tom
On Tue, Jun 23, 2020, 6:06 AM I?aki Ucar <iucar at fedoraproject.org> wrote:

            

  
  
#
Maybe Elliott?
On Tue, 23 Jun 2020 at 15:01, Tom Callaway <spotrh at gmail.com> wrote:

  
    
#
On Tuesday, 23 June 2020 14.01.39 WEST Tom Callaway wrote:
What needs to be done and how can the work be streamlined?

I asked this since this procedure will happen for other updates (in one year I 
know but time flies).

I am a provenpackager so that I can drive the process.

Regards,
#
On Tue, 23 Jun 2020 at 17:01, Jos? Ab?lio Matos <jamatos at fc.up.pt> wrote:
Great, thanks! :) Basically, we need (correct me if I'm wrong, Tom):

1) A new user side tag.
2) For R, merge master in F32 and send a build to that side tag.
3) For all packages, either merge master into F32 or just increase the
release version and send builds to that side tag *in order*.
4) When everything is built, create an update from that side tag.

I have tooling in my Copr repo to resolve CRAN dependencies and create
a list of lists of builds in order. I can give you that. But note that
Bioc packages (and others) are not included.
#
On Tuesday, 23 June 2020 16.10.07 WEST I?aki Ucar wrote:
The part that worries me is the *in order*. :-)

Do we need to do bootstrap builds that are later replaced by the real 
builds?

Because other than that it is business as usual. :-)
#
There are a few of those, but not many.
On Tue, Jun 23, 2020 at 11:31 AM Jos? Ab?lio Matos <jamatos at fc.up.pt> wrote:

            

  
  
#
I could do so, but it wouldn't be until this weekend.

Also, Tom mixed in some version bumps to the rebuild, so we'd have to
check whether those would actually be okay in a released version. I
would also appreciate some review on new (mostly test) dependencies
for these bumps.
On Tue, 23 Jun 2020 at 09:43, I?aki Ucar <iucar at fedoraproject.org> wrote:

  
    
#
Thanks, Jos? and Elliott. I can help with reviews.

I attach here a list of batches of CRAN packages to be rebuilt in
order (batches separated by a blank line), and the script that
generates it. Hope it helps.

I?aki

On Wed, 24 Jun 2020 at 11:35, Elliott Sales de Andrade
<quantum.analyst at gmail.com> wrote:

  
    
#
Oh, and maybe in this process we could add to all packages the
requirement on R(ABI) = 4 that Tom implemented.
On Wed, 24 Jun 2020 at 11:42, I?aki Ucar <iucar at fedoraproject.org> wrote:

  
    
#
On Wed, 24 Jun 2020 at 05:42, I?aki Ucar <iucar at fedoraproject.org> wrote:
Thanks. I have a few open already:
- https://bugzilla.redhat.com/show_bug.cgi?id=1839451 - R-servr
- https://bugzilla.redhat.com/show_bug.cgi?id=1839456 - R-filelock
- https://bugzilla.redhat.com/show_bug.cgi?id=1839474 - R-DBItest

  
    
#
On Wednesday, 24 June 2020 10.42.10 WEST I?aki Ucar wrote:
Sure it does.
I am doing the rebuild in side tag f32-build-side-24797.
#
On Wednesday, 24 June 2020 10.44.14 WEST I?aki Ucar wrote:
For that we need to start with rawhide and then change the R-rpm-macros 
package.

Probably it should be enough to change the /usr/lib/rpm/R-deps.R script to add

Requires: R(ABI)=4.0

I suggest to continue this as is and then implement that change in rawhide and 
bring it back to Fedora 32 as new updates are issued.

What do you think?
#
This work is already complete in rawhide.

Tom
On Thu, Jun 25, 2020, 1:12 PM Jos? Ab?lio Matos <jamatos at fc.up.pt> wrote:

            

  
  
#
On Thursday, 25 June 2020 18.26.20 WEST Tom Callaway wrote:
OK.
Building R-rpm-macros now and then untag and rebuild the modules already done. 
After all this is an iterative procedure. :-(

Thank you for the remark. :-)
#
On Wednesday, 24 June 2020 10.42.10 WEST I?aki Ucar wrote:
R-acepack depends on R-testthat so I moved it the batch after R-testthat. :-)

If I find a circular dependency I will need to do a bootstrap build.

Fun lies ahead... :-)
#
On Thu, 25 Jun 2020 at 19:01, Jos? Ab?lio Matos <jamatos at fc.up.pt> wrote:
Thanks for starting off builds. However, please be careful merging to
master, as some packages were bumped and have incompatibilities that
should not be put in stable releases. I will try to come up with an
exact list soon.

  
    
#
On Friday, 26 June 2020 00.45.46 WEST Elliott Sales de Andrade wrote:
I am doing the process manually to ensure that this does not happen.

Basically my method is for a given package called appropriately pkg:

cd R-pkg
less R-pkg.spec # read file to ensure that it is safe to build it

if it is safe then I run:

$ fedpkg switch-branch f32 && git merge master && git push && fedpkg build --
target=f32-build-side-24797 && fedpkg switch-branch master; cd ..

the idea, of course, is that if the merge fails then the case needs to be 
dealt manually.

FWIW most of the credit goes to Tom (spot) since the heavy lifting was already 
done. :-)

I will welcome your list as it becomes easier to find those cases. As I said 
before I am doing my best to avoid those cases.
#
On Thursday, 25 June 2020 18.26.20 WEST Tom Callaway wrote:
BTW I suspect that now we can simplify the requires generator.
As an example we have:

$ rpm -qp --requires R-AUC-0.3.0-8.fc32.noarch.rpm
R(ABI) = 4.0
R-core
rpmlib(CompressedFileNames) <= 3.0.4-1
rpmlib(FileDigests) <= 4.6.0-1
rpmlib(PayloadFilesHavePrefix) <= 4.0-1
rpmlib(PayloadIsZstd) <= 5.4.18-1

I guess that now R(ABI) and R-core are synonyms and thus we can remove the R-
core requirement.

Of course that this in minimal and mainly aesthetic. :-)
#
On Tuesday, 23 June 2020 16.48.53 WEST Tom Callaway wrote:
Hi Tom,
  I noticed that for example in R-assertthat you have used the bcond:

%bcond_with check

would not it be better to use bootstrap instead to take advantage of:
https://docs.fedoraproject.org/en-US/packaging-guidelines/#bootstrapping 

I am asking for curiosity since from now on we will need to do this dance once 
a year for the releases that we deem worth (I would expect that to be rawhide 
and the latest stable).

Forgive me if the question does not make sense since I am still trying to make 
sense of this maze. :-)
#
On Fri, 26 Jun 2020 at 11:36, Jos? Ab?lio Matos <jamatos at fc.up.pt> wrote:
I used bcond locally and wrongly assumed that fedpkg build would
support --with BCOND and --without BCOND. Instead, the way to activate
it is to change to "%bcond_with check" and then revert to
"%bcond_without check". The only difference with bootstrap is that
"bootstrap" is recognized and a suffix ~bootstrap can be added
automatically to the resulting build. So the question is whether we
want that suffix or not.
#
Hi Jos?,
On Thu, 25 Jun 2020 at 20:05, Jos? Ab?lio Matos <jamatos at fc.up.pt> wrote:
Ah, great, thanks for being careful.
For simple bumps that would not cause any merge issues, as it would
just fast-forward.
It turns out the list is shorter than I expected; I went through their
NEWS files to determine which ones are okay:

               master        f32
RCurl        1.98.1.2  1.95.4.12
RSQLite         2.2.0      2.1.2
biglm           0.9.2      0.9.1
digest         0.6.25     0.6.22
fs              1.4.1      1.3.1        - was missing dependencies
before, I think?
gargle          0.5.0      0.4.0         - was missing latest
testthat; should be fine to update
glue            1.4.1      1.3.2         - requires new vctrs; do not bump
haven           2.3.0      2.2.0         - requires new vctrs; do not bump
httpuv          1.5.4    1.5.3.1
later         1.1.0.1      1.0.0
lubridate       1.7.8      1.7.4         - should be okay to build
1.7.8, but not 1.7.9, as it needs new vctrs
matrixStats    0.56.0     0.55.0        - there are 'significant
changes', but I'm not familiar with this package for whether that's
okay
multcomp       1.4.13     1.4.12
mvtnorm         1.1.0     1.0.12
pdftools        2.3.1        2.2      - needs new dependencies, but I
think check was turned off
pkgload         1.1.0      1.0.2
prettyunits     1.1.1      1.0.2
qcc               2.7        2.2
quantities      0.1.4      0.1.3        - should be okay to build
0.1.4, but not 0.1.5, as it needs new vctrs
rgdal           1.5.8      1.4.8
rmarkdown         2.2        2.1
rsvg              2.1        1.3        - probably okay, but missing
test dependencies
sandwich        2.5.1      2.3.0
tidyr           1.1.0      1.0.2        - requires new vctrs; do not bump
tinytest        1.2.1      1.1.0
tinytex          0.23       0.20
vctrs           0.3.0      0.2.4        - many breaking changes; do not bump

Feel free to double-check these.

Also, let me know if you need to stop building and I can do so.

  
    
#
On Friday, 26 June 2020 10.47.13 WEST I?aki Ucar wrote:
AFAIU the issue is that the bootstrap scheme does not requires to bump the 
release because with the suffix (~) is considered lower while changing from 
"%bcond_with check" to "%bcond_without check" requires to bump the release.

The issue is that koji does not allow 2 builds with the same nvr.

Since we should ensure the upgrade path that also requires to bump the release 
number in rawhide before applying it on f32 (in this particular case).