New version of Rtools for Windows

The r.dll crash is easy - you need to be using gcc-ar for ar, and gcc-ranlib for ranlib. I also posted a patch to fix the check failure for stack probing, as lto optimizes away the stack probing code, as it should.

yes, lto build's speed gain is very impressive.

------------------------------
On Thu, Jan 8, 2015 2:01 PM GMT Henric Winell wrote:

On 2015-01-08 14:18, Avraham Adler wrote:

Very timely, as this is how I got into the problem I posted about
earlier; maybe some of the problems I ran into will mean more to the
you and the experts on this thread, Dr. Murdoch.For reference, I run
Windows 7 64bit, and I am trying to build a 64 bit version of R-3.1.2.

As we discussed offline, Dr. Murdoch, I've been trying to build R
using more recent tools than GCC4.6.3 prerelease. Ruben Von Boxen
(rubenvb) told me he is no longer developing his own builds of GCC,
but is focusing on MSYS2 and the mingw64 personal builds. So, similar
to what Jeroen said, I first installed MSYS2, whose initial
installation on windows is not so simple[1]. After the initial
install, the following packages need to be manually installed: make,
tar, zip, unzip, zlib, and rsync. I also installed base-devel, which
is way more than necessary, but there may be packages in there which
are necessary.

I originally installed the most up-to-date version of GCC (4.9.2)[2],
and I did pick the -seh version, as since I install (almost) all
packages from source (the one exception being nloptr for now), the
exception handling should be consistent and it is supposed to up to
~15% faster[3].

The initial build crashed with the following error:

gcc -std=gnu99 -m64 -I../../include -I. -DHAVE_CONFIG_H  -O3 -Wall
-pedantic -mtune=core2   -c xmalloc.c -o xmalloc.o
ar crs libtre.a regcomp.o regerror.o regexec.o tre-ast.o tre-compile.o
tre-match -approx.o tre-match-backtrack.o tre-match-parallel.o
tre-mem.o tre-parse.o tre-stack.o xmalloc.o
gcc -std=gnu99 -m64   -O3 -Wall -pedantic -mtune=core2   -c compat.c -o compat.o
compat.c:65:5: error: redefinition of 'snprintf'
  int snprintf(char *buffer, size_t max, const char *format, ...)
      ^
In file included from compat.c:3:0:
F:/MinGW64/x86_64-w64-mingw32/include/stdio.h:553:5: note: previous
definition of 'snprintf' was here
  int snprintf (char * __restrict__ __stream, size_t __n, const char *
__restrict__ __format, ...)
      ^
compat.c:75:5: error: redefinition of 'vsnprintf'
  int vsnprintf(char *buffer, size_t bufferSize, const char *format,
va_list args)
      ^
In file included from compat.c:3:0:
F:/MinGW64/x86_64-w64-mingw32/include/stdio.h:543:7: note: previous
definition of 'vsnprintf' was here
    int vsnprintf (char * __restrict__ __stream, size_t __n, const char
* __restrict__ __format, va_list __local_argv)
        ^
../../gnuwin32/MkRules:218: recipe for target 'compat.o' failed
make[4]: *** [compat.o] Error 1
Makefile:120: recipe for target 'rlibs' failed
make[3]: *** [rlibs] Error 1
Makefile:179: recipe for target '../../bin/x64/R.dll' failed
make[2]: *** [../../bin/x64/R.dll] Error 2
Makefile:104: recipe for target 'rbuild' failed
make[1]: *** [rbuild] Error 2
Makefile:14: recipe for target 'all' failed
make: *** [all] Error 2

After doing some checking (for example see [4]), I asked Duncan about
the problem, and he suggested moving the #ifndef _W64 in compat.c up
above the offending lines (65-75). That did not work, so, I figured
(it seems mistakenly from the other thread) that if those functions
are included from stdio already, I can just delete them from compat.c.
The specific lines are:

int snprintf(char *buffer, size_t max, const char *format, ...)
{
     int res;
     va_list(ap);
     va_start(ap, format);
     res = trio_vsnprintf(buffer, max, format, ap);
     va_end(ap);
     return res;
}

int vsnprintf(char *buffer, size_t bufferSize, const char *format, va_list args)
{
     return trio_vsnprintf(buffer, bufferSize, format, args);
}

Continuing the build using 4.9.2 crashed again at the following point:

gcc -std=gnu99 -m64 -I../include -I. -I../extra -DHAVE_CONFIG_H
-DR_DLL_BUILD  -O3 -Wall -pedantic -mtune=core2   -c malloc.c -o
malloc.o
windres -F pe-x86-64  -I../include -i dllversion.rc -o dllversion.o
gcc -std=gnu99 -m64 -shared -s -mwindows -o R.dll R.def console.o
dynload.o editor.o embeddedR.o extra.o opt.o pager.o preferences.o
psignal.o rhome.o rt_complete.o rui.o run.o shext.o sys-win32.o
system.o dos_wglob.o malloc.o ../main/libmain.a ../appl/libappl.a
../nmath/libnmath.a getline/gl.a ../extra/xdr/libxdr.a
../extra/pcre/libpcre.a ../extra/bzip2/libbz2.a
../extra/intl/libintl.a ../extra/trio/libtrio.a ../extra/tzone/libtz.a
../extra/tre/libtre.a ../extra/xz/liblzma.a dllversion.o -fopenmp -L.
-lgfortran -lRblas -L../../bin/x64 -lRzlib -lRgraphapp -lRiconv
-lcomctl32 -lversion
collect2.exe: error: ld returned 5 exit status
Makefile:150: recipe for target 'R.dll' failed
make[3]: *** [R.dll] Error 1
Makefile:179: recipe for target '../../bin/x64/R.dll' failed
make[2]: *** [../../bin/x64/R.dll] Error 2
Makefile:104: recipe for target 'rbuild' failed
make[1]: *** [rbuild] Error 2
Makefile:14: recipe for target 'all' failed
make: *** [all] Error 2

As all those files existed in their correct places, the only reason I
could think of that this would fail here is that GCC version 4.9 did
make some changes to enhance link-time optimization [5], and probably
something isn't compatible.

Right.  Just before Christmas, Hin-Tak Leung reported build failure with
LTO:

https://stat.ethz.ch/pipermail/r-devel/2014-December/070286.html
https://stat.ethz.ch/pipermail/r-devel/2014-December/070319.html


Many thanks to you and others for looking into this,
Henric

I then downgraded to GCC 4.8.4 [6], and,
with the deletion of those 10 or so lines from compat.c, I can
complete the build straight through rinstaller. However, I get that
failure issue due to the extra 0 in scientific notation [7].

It does not matter if I do the entire process in the MSYS2
environment, or if I do in in Windows with msys\usr\bin in my path.

Na?vely, it seems that if there were some what for stdio to be
included in compat.c, yet the versions of snprintf and vsprintf in
that file to "override" the standard, perhaps this method would work.
Of course, running make check-all may uncover more issues. I intend to
run the equivalent checks (from the tools library) inside of R with
kill on failure turned off to see if anything else is problematic.

Hopefully, something in this description resonates with one of the
readers here. If anyone has any ideas as to how to circumvent the
issues with compat.c, I'd be very grateful.

Thank you,

Avi


[1] http://sourceforge.net/p/msys2/wiki/MSYS2%20installation/
[2] http://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win64/Personal%20Builds/mingw-builds/4.9.2/threads-win32/seh/x86_64-4.9.2-release-win32-seh-rt_v3-rev1.7z/download
[3] https://stackoverflow.com/questions/15670169/what-is-difference-between-sjlj-vs-dwarf-vs-seh
[4] http://www.tt-forums.net/viewtopic.php?p=1034657&sid=613fa47a379ffaa0b9a9fb182a4180e3#p1034657
[5] https://gcc.gnu.org/gcc-4.9/changes.html
[6] http://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win64/Personal%20Builds/mingw-builds/4.8.4/threads-win32/seh/x86_64-4.8.4-release-win32-seh-rt_v3-rev0.7z/download
[7] https://stat.ethz.ch/pipermail/r-devel/2015-January/070354.html

Date: Wed, 07 Jan 2015 20:31:07 -0500

From: Duncan Murdoch <murdoch.duncan at gmail.com>
To: Jeroen Ooms <jeroenooms at gmail.com>
Cc: "R-devel at r-project.org" <r-devel at r-project.org>
Subject: Re: [Rd] New version of Rtools for Windows
Message-ID: <54ADDDDB.4020500 at gmail.com>
Content-Type: text/plain; charset=utf-8

On 07/01/2015 5:20 PM, Jeroen Ooms wrote:

On Wed, Jan 7, 2015 at 8:00 AM, Duncan Murdoch <murdoch.duncan at gmail.com> wrote:

This version includes only minor updates to the tools.  I indicated last summer that I was hoping to update GCC from the current version 4.6.3 before the R 3.2.0 release, but this now looks unlikely, unless someone else with experience building it can help.

I have been looking into this a bit over the past few months, also
with mixed success. Nevertheless, below some experiences that might be
worth sharing.

The guys from mingw-w64 recommended (quite strongly) to move away from
multilib. They explained that the standard approach is to create two
separate toolchains; one that targets win32 and the other one that
targets win64 (both tool chains can compiled for win32). Hence the
only difference for R would be that instead of passing "-m64" and
"-m32", it would need to set the path to the proper compiler.

There are several initiatives that provide very complete suites of
precompiled mingw-w64 tools. I think the ideal scenario would be if we
could take advantage of an existing tool chain as we do on other
platforms, although perhaps I do not fully understand the R-specific
requirements on the windows compiler.

I feel quite strongly that we need to be able to build the toolchain,
rather than relying on binaries produced by others.  We may need to
customize the toolchain, or we may need to rebuild it when a bug is
identified.  Lots of binary builders abandon their builds and you can't
count on them to solve problems at a later date.

One project that looks very promising is msys2 [1,2]. It has a package
manager (port of pacman from arch linux) and comes with a pretty
complete set of msys [3] and other [4] packages that seems quite well
maintained.

Do they post complete instructions for building?  That's what I'm
looking for.  I don't want to develop a build script (I don't know how),
but I would like to have one.

Duncan Murdoch

The only issue I ran into with msys2 is that it uses a different c++
exception model (seh/dwarf) than the current Rtools (which uses sjlj).
See also [5]. Therefore, if a library uses exceptions, we cannot use
the current Rtools to link a static library that was created with
msys2  [6]. I am not sure if it also be a problem the other way
around, and if this is still the case for recent versions of
gcc/mingw.

Finally, Ruby has build very similar to Rtools called DevKit-mingw64
[7] that we might be able to borrow from.


[1] https://msys2.github.io/
[2] http://stackoverflow.com/questions/25019057/how-are-msys-msys2-and-msysgit-related-to-each-other
[3] https://github.com/Alexpux/MSYS2-packages
[4] https://github.com/Alexpux/MINGW-packages
[5] http://stackoverflow.com/questions/15670169/what-is-difference-between-sjlj-vs-dwarf-vs-seh
[6] http://stackoverflow.com/questions/7751640/undefined-reference-to-gxx-personality-sj0
[7] http://rubyinstaller.org/downloads/