Skip to content

binary R packages for GNU/Linux

17 messages · Iñaki Ucar, Carl Boettiger, Lluís Revilla +7 more

Original
Tobias Verbeke
# 
L.S.

AFAICS the Writing R Extensions and R Installation and Administration manuals do not explicitly discuss binary R packages on GNU/Linux. Only installation from source is mentioned (https://cran.r-project.org/doc/manuals/R-admin.html#Installing-packages-1)
and when discussing repository layouts (https://cran.r-project.org/doc/manuals/R-admin.html#Setting-up-a-package-repository) no mention is made of conventions for GNU/Linux distributions.

The proprietary Package Manager (PPM) from Posit (https://packagemanager.posit.co/client/#/) does offer binary packages for GNU/Linux, but the usage of this service is restricted in ways that go against the principles of open source (https://posit.co/about/posit-service-terms-of-use/). By way of example, mirroring is not allowed and certain categories of users are excluded (age categories, competitors, ...). This is maybe expected to some, but this clearly does not offer a proper foundation for the distribution of open source R packages.

For this reason I am wondering whether the R project / CRAN would not be better placed and/or open to support distribution of binary R packages on GNU/Linux.

A second, orthogonal question is whether the R project can advance an official convention for the repository layout related to the distribution of binary GNU/Linux packages. Our proposal would be to use something along

http://mydomain.com/bin/linux/jammy/x86_64/contrib/4.4

which IMHO is more elegant than

http://mydomain.com/bin/linux/jammy-x86_64/contrib/4.4

(and which mimicks the documented MacOS convention

http://mydomain.com/bin/macosx/big-sur-x86_64/contrib/4.4).

Anyone?

Obviously willing to work out details and collaborate on the topic.

Kind regards,
Tobias
Simon Urbanek
# 
Tobias,

although we did discuss the possibility of extending the os/toolchain/architecture notation for binary packages beyond macOS, Linux was not necessarily on the list as Linux distributions have already established ways of providing binaries, so it does not seem productive to duplicate the effort. Can you elaborate a bit more on what you had in mind? Binaries are by design specific to toolchain, distribution and architecture, so there is no such thing as a "GNU/Linux binary". The only reliable way to distribute packages in Linux is from sources or by the Linux distribution repositories. Binaries are inherently tied to system dependencies and their versions, so such concept doesn't make any sense outside of the distribution. There is no such thing as a "jammy binary" to take up your example - it would have to depend on the distribution, toolchain and all library versions as well.

Cheers,
Simon
Tobias Verbeke
# 
Hi Simon,

Thank you for your prompt reply.

----- Original Message -----
We agree. It is just used as a generic term here to denote a package built for a specific version of a distribution for a specific architecture and its toolchain.
jammy is a specific version of the distribution (Ubuntu 22.04 LTS) and the architecture is included in my proposal (x86_64)

http://mydomain.com/bin/linux/jammy/x86_64/contrib/4.4

In my personal experience (~25y of GNU/Linux, mostly Debian, Ubuntu and CentOS) versions of the toolchain will not differ in a practically relevant way within a particular version of a distribution, so it is possible to build and distribute packages for a specific version of a distribution on a specific architecture for a specific series of R (4.4). I guess that if it was not possible, the effort would also not have been undertaken by PPM mentioned earlier. They offer (I skip the non-publicly available Linux distros) CentOS 7 (centos7), Rocky Linux 9 (rhel9), Ubuntu 20.04 (focal), Ubuntu 22.04 (jammy), Ubuntu 24.04 (noble), Debian 11 (bullseye), Debian 12 (bookworm). Another argument to demonstrate the feasibility is the r2u project (https://github.com/eddelbuettel/r2u). It offers CRAN as Ubuntu Binaries, but in order to build these Ubuntu Binaries it actually makes use of the binary R packages built by PPM. Quoting from https://eddelbuettel.github.io/r2u/: "For the CRAN binaries we either repackage P3M/RSPM/PPM builds (where available) or build natively." They cover all CRAN packages. The usage of PPM as a source is, of course, a weakness (in the grand scheme of things), but the point here is about the feasibility of building the packages in a portable way per version of a particular distribution, architecture etc.

I do understand arguments pro 'distribution binaries' (e.g. dependency resolution of system dependencies), but there are also arguments pro 'CRAN binaries' (binary builds of the R packages), since it can be convenient to allow for fast installation of arbitrary R packages without the need of more general sudo rights (required for installation of distribution binaries).

If R-core/CRAN maintainers agree that the answer to my first question is 'no', I am fine with that, but I can only know when asking.

It still leaves the answer to my second question ('official' repository layout conventions) open. It could be: 'we think it is a bad idea, so don't propose a structure' or 'we think it is a bad idea, but propose the following structure for people with bad ideas' :-).

Kind regards,
Tobias
Dirk Eddelbuettel
#
On 10 February 2025 at 11:00, Tobias Verbeke wrote:
| Another argument to demonstrate the feasibility is the r2u project
| (https://github.com/eddelbuettel/r2u). It offers CRAN as Ubuntu Binaries, but
| in order to build these Ubuntu Binaries it actually makes use of the binary R
| packages built by PPM. Quoting from https://eddelbuettel.github.io/r2u/: "For
| the CRAN binaries we  either repackage P3M/RSPM/PPM builds (where
| available) or build natively." They cover all CRAN packages. The usage of PPM
| as a source is, of course, a weakness (in the grand scheme of things), but
| the point here is about the feasibility of building the packages in a
| portable way per version of a particular distribution, architecture etc.

As you brought this up, allow me to clarify: The re-use (where possible) is
simply a shortcut "where possible".  Each day when I cover updated packages,
I hit maybe 5 per cent of packages where for reasons I still cannot decipher
p3m.dev does not have a binary, so I build those 5 per cent from source.
Similarly for the approx 450 BioConductor packages all builds are from
source.

Rebuilding everything from source "just because we want to" is entirely
possible but as it is my time waiting for binaries I currently do not force
full rebuilds but I easily could. Also note that about 22% of packages
contain native code, leaving 78% which are not. Re-use is even simpler there
as these 78% as they contain only (portable) R processing. So if we wanted to
compile all native packages for Ubuntu, we could. It is a resourcing issue
that has not yet been a prioruty for me. Inaki does it for Fedora, Detlef
does it for OpenSUSE.

The more important point of these package is the full system integration. You
do get _all_ binary dependencies declared, exactly as a distribution-native
package (of which Debian/Ubuntu have a bit over 1k) would. Guaranteed.
Reliably. Fast. That is a big step up for large deployments, for testing, for
less experienced users.

So thanks for starting a discussion around this as 'we' as a community are
falling a bit short here. One open question is if we could pull something off
that works like the Python wheels and offers cross-distro builds, ideally
without static linking. Your "CRAN libraries" added to the ld.so path may do
this. I do not know how feasible / involved this would be so so far I
concentrated on doing something simpler -- but feasible and reliable by
working exactly as the distribution packages work.

All that said, thanks for the starting this discussion!

Cheers, Dirk
Iñaki Ucar
#
On Mon, 10 Feb 2025 at 14:09, Dirk Eddelbuettel <edd at debian.org> wrote:
And for completeness, [1] is where we painstakingly* maintain a list
of system dependencies, [2] is where the daily magic happens for
keeping track of CRAN, and [3] performs the heavy-lifting and
publishes an RPM repository with the result.

[1] https://github.com/cran4linux/sysreqs
[2] https://github.com/cran4linux/cran2copr
[3] https://copr.fedorainfracloud.org/coprs/iucar/cran

*Because, you know, SystemRequirements.
Indeed, thank you, Tobias.
It would be perfectly feasible to maintain sync'ed builds (in terms of
version) of system dependencies at CRAN-provided (RPM, APT...)
repositories as compat packages for various distributions, then all
packages could be built once and shipped everywhere (i.e. cross-distro
builds). Collaterally, this would increase reproducibility of package
checks to a certain extent.

I offered my help in these matters in the past, but was kindly
declined. That hand remains extended.

Best,
I?aki

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Carl Boettiger
    

    

      
      #
    

  


  

      
Great discussion.

Just to note another example I don't think was mentioned -- The r-universe
project also builds binaries for Linux (Ubuntu latest)
https://docs.r-universe.dev/install/binaries.html (as well as other targets
including wasm).  It also provides binaries for Bioconductor and packages
on any git-based version control platform (e.g. GitHub).

R Universe is open source and a top-level project of the R Consortium.

Cheers,

Carl

---
Carl Boettiger
http://carlboettiger.info/

        
On Mon, Feb 10, 2025 at 5:30?AM I?aki Ucar <iucar at fedoraproject.org> wrote:

        

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

        

  
  


  


      

    

    

      
      

        


  

        

      Lluís Revilla
    

    

      
      #
    

  


  

      
Adding a different project to this discussion:

Recently at the R repository working group a project to provide them for
multiple distributions and toolchain was presented.
See the documentation about what do they (Patrick Schratz) share:
https://docs.r-package-binaries.devxy.io/index.html

The existence of these projects (Alpine, RedHat, Fedora, Ubuntu, r2u, P3M,
r-universe, ...) providing binaries show there is interest even if there
are multiple combinations of platforms, toolchains and library versions.
If these projects cooperate between them (as shown by r2u project) and with
CRAN, I think this will reduce duplicated effort and help the users (and
repositories).
Binaries could be served in multiple places matching what R requires and/or
what the distribution uses without much more (technical) effort.

Cheers,

Llu?s

        
On Mon, 10 Feb 2025 at 16:35, Carl Boettiger <cboettig at gmail.com> wrote:

        

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

        

  
  


  


      

    

    

      
      

        


  

        

      Dirk Eddelbuettel
    

    

      
      #
    

  


  

      
On 10 February 2025 at 07:35, Carl Boettiger wrote:

        
| Great discussion.??
| 
| Just to note another example I don't think was mentioned -- The r-universe
| project also builds binaries for Linux (Ubuntu latest) https://
| docs.r-universe.dev/install/binaries.html (as well as other targets including
| wasm).? It also provides binaries for Bioconductor and packages on any
| git-based version control platform (e.g. GitHub).??

Yes ... but these are 'naked' binaries as created by 'R CMD INSTALL --build'
but without system integration (and as such mirror what p3m.dev does). This
has its merits (it is simpler, can cover more OS variants) but it is also
more limited.

What we (ie Detlef, Inaki, myself) myself do for the distros is fundamentally
different. Both are merits, both can coexist, but I like the added 'oomph'
you get by integrating properly with the distribution you deploy on.  Ubuntu
is a pretty useful base case.

Dirk

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Hadley Wickham
    

    

      
      #
    

  


  

          

      
      
      
    

        
If this terms of service is the main sticking point, I could certainly find
out if we could change them. From a quick read, my personal sense (with no
internal exploration whatsoever) is that these (a) were probably written by
a lawyer with no specific understanding of PPM, let alone that these are
binaries built from open-source packages, and (b) probably aren't legally
binding anyway (since for most PPM usage, people downloading packages never
see the TOS).

Hadley

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Simon Urbanek
    

    

      
      #
    

  


  

          

      
      
      
    

        
In case it wasn't clear - precisely this was my point, and I was counting on all those doing the hard work already like Dirk to speak up (thanks, Dirk and I?aki). I'm not convinced that "naked" binaries are that useful, so just creating a new subdirectory isn't a solution IMHO. It works for some cases, but not in general - many can build their specific binaries, but it doesn't mean they work for others. The real work has to be done by people that have experience like Dirk and others mentioned above, not CRAN. Any such binaries are tied to the R build (just like macOS CRAN R sets its platform/arch target so it can use the CRAN binaries), so it requires cooperation across the spectrum including the distros. That said, the Linux maintainers are certainly invited to liaise with CRAN if they think it would be beneficial to host things on CRAN - we have done in the past, but with the explosion of distros and releases the trend was to go do the distros instead.

Cheers,
Simon

  
  


  


      

    

    

      
      

        


  

        

      Jeroen Ooms
    

    

      
      #
    

  


  

      
On Mon, Feb 10, 2025 at 9:48?PM Simon Urbanek

        
<simon.urbanek at r-project.org> wrote:

            

      
      
      
    

        
The "naked binaries" are widely used, and therefore probably useful to
many folks, me included. Some standardisation on paths would be
incredibly useful, even if CRAN would not offer such binaries, other
repositories could.

Note that Dirks apt packages are literally repackaged binaries form
p3m, for convenience of ubuntu (root) users. As Dirk noted, some
people prefer installing binaries via apt rather than
install.packages(), which is all fine, but methods both have pros and
cons. However, any arguments against p3m based on TOS or compatibility
will extend to the repackaged apt binaries, so I don't think that
argument holds.

  
  


  


      

    

    

      
      

        


  

        

      Dirk Eddelbuettel
    

    

      
      #
    

  


  

      
On 10 February 2025 at 23:19, Jeroen Ooms wrote:

        
| The "naked binaries" are widely used, and therefore probably useful to
| many folks, me included. Some standardisation on paths would be
| incredibly useful, even if CRAN would not offer such binaries, other
| repositories could.

Sure, but naked binaries can (and do) break more often i.e. real example from
just last week in another project p3m served a stringr dependency via a
stringi binary with a wrong libicu* outside of the particular release,
breaking.  That simply cannot and hence will not happen at r2u.

But choice is good. If people want to manually manage their system
dependencies they surely, they can also outsource it to another layer. As you
say, all trade-offs and if you are happy with naked binaries more power to
you. I prefer the proper integration and that's what I built. 

Dirk

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Iñaki Ucar
    

    

      
      #
    

  


  

      
On 10 February 2025 at 23:19, Jeroen Ooms wrote:

            

      
      
      
    

        
Some people prefer having all their binaries *managed* by apt/dnf, but
still using install.packages() to trigger that work. I count myself
there, and that's what I built.

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Stefan Evert
    

    

      
      #
    

  


  

          

      
      
      
    

        
Just a +1 on this point and many thanks to Dirk for providing r2u, which we happily use to install R packages on our Linux servers. 

? Stephanie

  
  


  


      

    

    

      
      

        


  

        

      Tim Taylor
    

    

      
      #
    

  


  

      
On Tue, 11 Feb 2025, at 8:07 AM, I?aki Ucar wrote:

            

      
      
      
    

        
Ditto. It really is automagical!

FWIW - I think this integration with the distributions is key to a painless experience. Perhaps a better question is to ask Dirk, I?aki et al. what their pain points are and whether they need any additional assistance? E.g. I'm aware of discussions around System Requirements that could help:

https://bugs.r-project.org/show_bug.cgi?id=18586

Tim

  
  


  


      

    

    

      
      

        


  

        

      Tobias Verbeke
    

    

      
      #
    

  


  

      
Thank you, Hadley. 

It would be great indeed if this can be fixed!

Kind regards,
Tobias

----- Original Message -----

            

      
      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Tobias Verbeke
    

    

      
      #
    

  


  

      
----- Original Message -----

            

      
      
      
    

        
Thank you, all, for the valuable input and reactions. Let me summarize.

I. choice is good 

Many people find 'naked binaries' useful and many people find 'distribution binaries' useful. In our own work, we see a myriad of R deployments and installations with very different processes in place and in certain cases 'naked binaries' are more convenient than 'distribution binaries'. In other cases, it is the other way around. Both deserve their place in the R world.

II. outsource to another layer

In case of 'naked binaries' dependency management indeed needs to be outsourced to another layer. People typically use either a data store (e.g. the Github repo I?aki mentioned - https://github.com/cran4linux/sysreqs or others e.g. https://github.com/r-hub/r-system-requirements). Other people add a REST API on top, which is e.g. what the PPM does (sysreqs resources in https://packagemanager.posit.co/__api__/swagger/index.html#/). To give a taste of how outsourcing to another layer looks like: to build reproducible environments we automatically generate Dockerfiles starting from the R code of a project. During the process of generating the Dockerfile, the package dependencies of the relevant packages are queried (using one of the sources above) and used to automagically insert the appropriate 'apt install' commands above and prior to the relevant install.packages commands in the Dockerfile.

III. goal in itself or ingredient for distribution packages

The building of 'naked binaries' are a necessary step in the process of building the 'distribution packages'. To some the naked binary is the end goal (often used in combination with a data source on dependencies). To others it is part of the ingredients for the end goal (distribution package). In both scenarios, it makes sense to store these naked binaries somewhere in order not to duplicate the builds. I personally think CRAN would be a good place to store these since it would be a neutral, central place that immediately gives a seal of quality and trust.

IV. repository layout

Many organizations have internal repositories mimicking a CRAN layout. Also many tool developers make certain assumptions on the repository layout. E.g. if one wants versioned package installs of source packages, one goes to src/contrib/Archive/$PACKAGE_NAME/ to retrieve the version. This is not explicitly documented and I am glad to hear Jeroen thinks such standardization would be
incredibly useful. Currently, the hosting of naked binaries actually makes use of a trick to put the packages under /src/contrib as in https://cran.r-universe.dev/bin/linux/noble/4.4/src/contrib/BayesERtools_0.2.0.tar.gz
While handy, it is a bit odd and may be confusing to some. We can further work out the specs, but a generalization of the repository layout that allows to also have an Archive for binary packages (for Windows, MacOS, naked binaries) would be very useful to be able to have fast versioned package installation. This is important if you want to combine performance and reproducible environments (which are impossible to achieve with the latest binary versions - even when using distribution binaries). This is not a request for CRAN to offer all versions of binary packages, but if the layout is clearly specified, other package manager infrastructure (public or internal to organizations) can fill this gap.

My message makes me think of the New Year Wish List mails Gabor Grothendieck tended to send out ~two decades ago (and maybe more recently) and I hope it triggers further discussion and ideas.

Thanks again!

Kind regards,
Tobias