Skip to content

Update on rtools4 and ucrt support

11 messages · Jeroen Ooms, jan Vitek, Duncan Murdoch +6 more

Original
Jeroen Ooms
# 
Hi all,

I received some questions this week about rtools4 (the windows
compiler bundle) in particular regarding support for ucrt, so below a
brief summary of the status quo:

As of May 2021, rtools4 has full support for ucrt. The toolchain
configuration is based on the upstream msys2 configuration, which are
very stable, and widely used by other open source projects as well as
the mingw-w64 developers themselves. The latest builds of rtools4 now
contain 3 toolchains:

 - c:\rtools40\mingw32: the 32-bit gcc-8-3.0 toolchain used as of R 4.0.0
 - c:\rtools40\mingw64: the 64-bit gcc-8-3.0 toolchain used as of R 4.0.0
 - c:\rtools40\ucrt64: a new 64-bit gcc-10.3.0 toolchain targeting ucrt

The total install size is about 1gb. Hence, if R were to switch to
ucrt at some point, users and sysadmins that have installed rtools4
after May 2021 are already equipped with proper toolchains for
building packages for both R 4.0+ as well as a potential ucrt versions
of R.

As before, for each of these toolchains, all extra libraries needed by
CRAN packages can easily be installed in rtools4 through pacman [1].
All system libraries in rtools-packages [2] have ucrt64 binaries [3].
When users contribute an update or a new rtools package, the CI
automatically builds and checks binaries for each of the above
toolchains, e.g [4]. The process is 100% automatic, transparent, and
reproducible. This provides a degree of accountability, and makes it
easy for R package authors to suggest improvements for the C/C++
libraries that they depend on (many have done so in the past 2 years).

Rtools4 is preinstalled on major CI/cloud services such as GitHub
actions. Popular open-source projects such as Apache-Arrow and TileDB
are already using the rtools4 toolchains to automatically build and
test their C++ libraries, as well as R bindings, for each commit, on
all target architectures (including ucrt64). Any R package author can
use the the same free services to check their packages on all compile
targets using rtools4 toolchains [5]. The r-devel CI tool on
https://r-devel.github.io checks every commit to base-R using ucrt64
toolchain from rtools4, which has proven to be very stable.

I am also aware that Tomas Kalibera also provides alternative
"experimental ucrt toolchain": a 6gb tarball with manually built
things on his personal machine. It is unclear to me why it was decided
to take this approach; it is certainly not needed to support ucrt
(ucrt is literally one flag in the toolchain configuration).
Fortunately, the ucrt tooclchains from rtools4 and Tomas Kalibera use
the same version of mingw-w64 and gcc, and are fully compatible, so
package authors could still use the rtools4 ucrt compilers icw the
R-devel-ucrt version that was built using this experimental toolchain
[5].

We spent an enormous amount of effort in the past years standardising
the Windows build tooling, and making the infrastructure automated,
open and accessible, such that everyone can learn how this works and
get involved. Many people have. Today if you install R and Rtools4 on
Windows, things "just work", regardless of whether this is a student
laptop, university server, or online CI system. Anyone can build R
packages, base-R, or any of the system libraries, following the steps,
and using standard tooling that other open source projects use. I
think it would be a big step back of R-core decides to go back to a
black-box system that is so opaque and complex that only one person
knows how it works, and would make it much more difficult for
students, universities, and other organisations to build R packages
and libraries on Windows.

Jeroen


[1] https://github.com/r-windows/docs/blob/master/rtools40.md#readme
[2] https://github.com/r-windows/rtools-packages
[3] https://cran.r-project.org/bin/windows/Rtools/4.0/
[4] https://github.com/r-windows/rtools-packages/pull/221
[5] https://github.com/r-windows/docs/blob/master/ucrt.md
3 days later
jan Vitek
# 
Hi Jeroen,

I mostly lurk on this list, but I was struck by your combative tone.

To pick on two random bits:
Tomas? tool chain isn't a blackbox, it has copious documentation (see [1]) 
and builds on any machine thanks to the provided docker container.

This is not to criticise your work which has its unique strengths, but to
state the obvious: these strengths are best discussed without passion 
based on factually accurate descriptions.


[1] https://developer.r-project.org/Blog/public/2021/03/12/windows/utf-8-toolchain-and-cran-package-checks/index.html


-jan

Jan Vitek, Professor 
Northeastern University
Duncan Murdoch
#
On 23/08/2021 8:15 a.m., jan Vitek via R-devel wrote:
I agree with Jan.  I'm not sure a discussion in this forum would be 
fruitful, but I really wish Jeroen and Tomas would get together, aiming 
to merge their toolchains, keeping the best aspects of both.

I haven't been involved in the development of either one, but have been 
a "victim" of the two chain rivalry, because the rgl package is not easy 
to build.  I get instructions from each of them on how to do the build, 
and those instructions for one toolchain generally break the build on 
the other one.  While it is probably possible to detect the toolchain 
and have the build adapt to whichever one is in use, it would be a lot 
easier for me (and I imagine every other maintainer of a package using 
external libs) if I just had to follow one set of instructions.

Duncan Murdoch
iuke-tier@ey m@iii@g oii uiow@@edu
#
On Mon, 23 Aug 2021, Duncan Murdoch wrote:

            

      
      
      
    

        
Here are just a few comments from my perspective (I am an R-core
member, but am not part of the CRAN team and do only very limited work
on Windows). Other R-core members may have different perspectives and
insights.

One bit of background: dealing with encoding issues on Windows has
been taking an unsustainable amount of R-core resources for some time
now. Tomas Kalibera has been taking the lead on trying to address
these issues in the existing framework, but this means he has not had
the time to make any of the many other valuable and important
contributions he could make. The only viable way forward is to move to
a Windows tool chain that supports UTF-8 as the C library current
encoding via the Windows UCRT framework.

Tomas Kalibera has, on behalf of all of R core and in
coordination with CRAN, been looking for a way forward for some
time and has reported on the progress in several blog posts at
https://developer.r-project.org/Blog/public/. This has lead to
the development of the MXE-based UCRT tool chain, which is now
well tested and ready for deployment.  Checks using the UCRT tool
chain have been part of the CRAN check process for a while. I
believe CRAN plans to switch R-devel checks and builds to the
UCRT tool chain during the upcoming CRAN downtime. I expect there
will be some communication from CRAN on this soon, including on
any issues in supporting binaries for both R-devel and R-patched.

In putting together something as large as a tool chain there will
always be many choices, each with advantages and disadvantages.  Some
things may be advantages in some settings and not others. Taking just
one case in point: Cross compilation. This is likely to be a better
approach for CRAN in the future and is supported by the MXE framework
on which the new tool chain is based.

The much more recent changes in rtools4 to support UCRT are at this
point not yet as well tested as the new tool chain. Once these changes
to rtools4 mature, and if binary compatibility can be assured, then
having a second tool chain may be useful in some cases.  But if there
are incompatibilities then it will be up to rtools4 to keep up with
the tool chain used by CRAN. On the other, contributing to improving
the MXE-based tool chain may be a better investment of time.

Best,

luke

            

      
      
      
    

        

      
      
    

        

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Avraham Adler
    

    

      
      #
    

  


  

      
On Mon, Aug 23, 2021 at 11:09 PM <luke-tierney at uiowa.edu> wrote:

        

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        
Thank you, Dr. Tierney. However, I am concerned about the not-insignificant
number of us who for various reasons can only do our development on
Windows. Rtools has been the official tool chain with which to build
windows for the at least 20 years I have been using R (yes, a babe in the
woods compared to most, but not a complete neophyte). Duncan and Jereoen
have each done yeoman?s jobs in ensuring that R can be built from source on
Windows and that packages can be developed for all OSS on Windows?even
Solaris SPARC.

I am much less aware of Thomas?s work, and I?ll gladly take the blame for
it, but I haven?t seen an accessible tool chain built by him which would
allow me, the Windows developer, to build R and all packages from source on
a native Windows box. Have I just missed it? If not, is that planned? If
R-core switches the official Windows toolchain, where does that leave us?

Thank you,

Avi

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Simon Urbanek
    

    

      
      #
    

  


  

      
Avi,

please see the announcement:

https://developer.r-project.org/Blog/public/2021/03/12/windows/utf-8-toolchain-and-cran-package-checks/index.html <https://developer.r-project.org/Blog/public/2021/03/12/windows/utf-8-toolchain-and-cran-package-checks/index.html>

the documentation is in

https://svn.r-project.org/R-dev-web/trunk/WindowsBuilds/winutf8/ucrt3/howto.html <https://svn.r-project.org/R-dev-web/trunk/WindowsBuilds/winutf8/ucrt3/howto.html>

Cheers,
Simon

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

        

  
  


  


      

    

    

      
      

        


  

        

      Avraham Adler
    

    

      
      #
    

  


  

      
On Tue, Aug 24, 2021 at 12:47 AM Simon Urbanek <simon.urbanek at r-project.org>
wrote:

            

      
      
      
    

        

      
      
    

        
Thank you, Simon. That was valuable. Skimming that quickly, I get a bit
concerned. I?ve been building from source and then using OpenBLAS in my R
source for many, many years now, and it looks like its support is tenuous
in the experimental chain. Similarly with packages like nloptr, where I
build NLOPT 2.6+ and have adjusted Jelmer?s code for it to work in R for
Windows. I maintain packages with Fortran/OpenMP and Rcpp(Parallel). I hope
that should the decision be made to switch, it will be done when the build
process is more streamlined, especially for some fundamental packages.

That being said, I must take the opportunity again to thank R-core, Tomas,
Jeroen, Duncan among many others who have built an infrastructure that
allows amateur programmers to contribute to the statistical infrastructure.

Thanks again,

Avi

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Tomas Kalibera
    

    

      
      #
    

  


  

      
On 8/24/21 12:03 AM, Avraham Adler wrote:

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        
Hi Avi,

if your R package includes source code (C/C++/Fortran), there is no 
fundamental difference, just the version of the compilers and/or 
external libraries may have changed.

nloptr checks are fine, see 
https://cran.r-project.org/web/checks/check_results_nloptr.html

The architecture is r-devel-windows-x86_64-gcc10-UCRT.

Best
Tomas

            

      
      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Simon Urbanek
    

    

      
      #
    

  


  

      
Avi,

thanks. Yes, the whole point of the developer blog posts by R-core are for uses to provide feedback, so that's great - it's odd that it required a somewhat orthogonal post to start the discussion several months later, but I'm glad we got here.

Note that the point of the switch is to iron out all issues that may be encountered and provide real CRAN testing - that's why both R-core and CRAN is involved in all this. This only affects R-devel since we want to be ready for the R 4.2.0 release, it won't affect the current release builds.

Anyway, now I'll leave it to the Windows experts to address the details and work together.

Cheers,
Simon

            

      
      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Dirk Eddelbuettel
    

    

      
      #
    

  


  

      
As I type this, we are eight messages into this thread -- but I am not sure
it has been made clear what the actual contentious issues are.

There appear to be two toolchains, and they appear to be interoperate (though
Duncan stated he had issues with an (arguably demanding) package).  Now, I
have the opposite (hence positive) experience.  For one package I look after,
a colleague took care of the (complicated in that case) 'needed to build the
package' pre-requirements by ensuring we have a UCRT variant.  Jeroen then
(unprompted) supplied a two-line/two-file PR to enable a Windows UCRT build
(piggy-backing on the existing Windows build), and with that the 'ERROR' I
had at CRAN reports under Tomas UCRT entry is gone. Net-net, this looks like
a working setup to me which combines both toolchains without issues.

And I was able to repeat this with a few more packages of mine for which
Jeroen's winlibs factory has libraries---these now build to under Tomas's
builder at CRAN.  So maybe this is not an either-or discussion?  So if there
are issues, could we be told what they are, and could we possibly help Jeroen
and Tomas to iron them out?

Dirk

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Gabriel Becker
    

    

      
      #
    

  


  

      
Hi all,

I will preface this with the fact that I don't do work on windows and the
following is based on remembered conversations/talks/etc from a while ago
so may be either incorrect or out of date, but I recall one of the major
things Jeroen was  targeting was use of/integration with a meaningful
package manager for external library dependencies in windows from-source
package builds, and that *I think* this was a part of his (then explicitly
experimental) Rtools4 setup (?)

Is the above correct, and if so, is there also package manager
integration/usage in Tomas' official R-core UCRT toolchain? If not, could
it be (perhaps, as Duncan suggested, via collaborative effort involving
Jeroen as well)?

I admit, both windows toolchains/builds and non-latin encodings are things
I have so far stayed away from, so I can't really contribute beyond that,
other than to say, as others have, that I do think both are impressive
pieces of work and that both Jeroen and Tomas should have our thanks thanks
for this and a lot of other work they put into R and the R community.

Best,
~G

        
On Mon, Aug 23, 2021 at 4:02 PM Dirk Eddelbuettel <edd at debian.org> wrote: