Hi all, Some bad news and then good news on the developments of the new tool chain for Windows. Bad news: Over the past week, Kevin and me have found several more problem cases in which the i686-493-win32-sjlj-rt_v3-s tool chain would cause R to freeze or segfault in the case of c++ exceptions, even when compiling with --no-asynchronous-unwind-tables. These happened mainly within Rcpp modules, but we also found a way to trigger a segfault without Rcpp [1]. We investigated many workarounds and tried patches for setjmp.h and psignal.h, but each solution introduced more problems then it solved. The good news: the mingw-w64 authors suggest to move away from the SJLJ exception model, and instead use DWARF on win32 and SEH on win64. These exception models are the default in mingw-w64 since gcc 4.8 because they are more reliable and performant. So we built a copy of i686-493-posix-dwarf-rt_v3-s and x86_64-493-posix-seh-rt_v3-s and used this to compile R. The results are quite amazing: both R and Rcpp pass all their checks without any special compiler flags (even without --no-asynchronous-unwind-tables). So far I have not been able to reproduce a single problem case with this new build, and it is noticeably faster as well. This confirms the sentiment of the mingw-w64 authors that SJLJ does not play nice with recent features of gcc. A zip file containing these new tool chains (unzip to c:/rtools) as well as a recent version of R-devel with DWARF/SEH (dubbed R-experimental) are available from [2]. I used icu-55.1 from [3] to build R with this new tool chain because the ICU version from Brian Ripley was built for SJLJ. Because ICU is the only c++ library required to build R, all of the other libraries (multi320.zip, Tcl, Cairo, libcurl) can still be taken from cran/rtools. So now we are torn on whether we should move away from SJLJ for the new tool chain. The advantages of DWARF/SEH are obvious: - More reliable - Better performance - Better tested because it is the default - Compatibly with GCC 5.x (for future updates) - We can use msys2 to build libraries The disadvantages: - No multilib support: requires separate toolchains for win32 and win64. - Requires a rebuild of existing C++ static libraries. - Windows build servers will need two separate sets of static c++ libs: one set built with SJLJ for the old tool chain (R 3.2) and one set built with DWARF/SEH for the new tool chain (R 3.3). Mixing them up will lead to linking errors. The situation is obviously suboptimal. I am especially concerned about the last point, because it important that we get CRAN on board, and this will definitely cause some temporary disruption. However it looks like this is the only way forward. Obviously if we are going to migrate I can help out with the rebuilding process. The good thing is that once we bite the bullet and make the switch, R on windows will be more prepared for the future. A major advantage of using the default mingw-w64 exception model is that we can use the excellent msys2 (the next iteration of cygwin/msys) to build c/c++ libraries for windows. In fact, we might be able to grab DWARF/SEH builds of most open source libraries straight from their repositories [3] as I did for ICU. We could even write an msys2 package [4] for R itself to work towards a more formal, automated build process on Windows similar to Linux. Jeroen [1] https://github.com/kevinushey/RToolsTest/commit/0471218b1e79d2cfd9477299dc1ba5f8213e9f21 [2] http://www.stat.ucla.edu/~jeroen/mingw-w64/ [3] http://repo.msys2.org/mingw/ [4] https://github.com/Alexpux/MINGW-packages
Windows toolchain update: seh vs sjlj
4 messages · Jeroen Ooms, Henric Nilsson (Public), Avraham Adler
2 days later
On Sun, Sep 13, 2015 at 12:36 PM, Jeroen Ooms <jeroen.ooms at stat.ucla.edu> wrote:
The good news: the mingw-w64 authors suggest to move away from the SJLJ exception model, and instead use DWARF on win32 and SEH on win64. These exception models are the default in mingw-w64 since gcc 4.8 because they are more reliable and performant.
Forgive the possibly ignorant question, but is there a reason not to use SEH for both 32- and 64-bit Windows? Avi
Den 2015-09-16 kl. 03:28, skrev Avraham Adler:
On Sun, Sep 13, 2015 at 12:36 PM, Jeroen Ooms <jeroen.ooms at stat.ucla.edu> wrote:
The good news: the mingw-w64 authors suggest to move away from the SJLJ exception model, and instead use DWARF on win32 and SEH on win64. These exception models are the default in mingw-w64 since gcc 4.8 because they are more reliable and performant.
Forgive the possibly ignorant question, but is there a reason not to use SEH for both 32- and 64-bit Windows?
Unfortunately, it has not been implemented for 32-bit yet: http://mingw-w64.org/doku.php/contribute Henric
Avi
_______________________________________________ R-SIG-windows mailing list R-SIG-windows at r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-windows
2 days later
On Sun, Sep 13, 2015 at 12:36 PM, Jeroen Ooms <jeroen.ooms at stat.ucla.edu> wrote:
Hi all, The good news: the mingw-w64 authors suggest to move away from the SJLJ exception model, and instead use DWARF on win32 and SEH on win64. These exception models are the default in mingw-w64 since gcc 4.8 because they are more reliable and performant. So we built a copy of i686-493-posix-dwarf-rt_v3-s and x86_64-493-posix-seh-rt_v3-s and used this to compile R. The results are quite amazing: both R and Rcpp pass all their checks without any special compiler flags (even without --no-asynchronous-unwind-tables). So far I have not been able to reproduce a single problem case with this new build, and it is noticeably faster as well. This confirms the sentiment of the mingw-w64 authors that SJLJ does not play nice with recent features of gcc.
Just an update on some testing. I've built R-devel_2015-09-10 on an i7-3740QM @ 2.7Ghz with 8MB RAM running Windows7 64bit under the new toolchain (4.9.3-SEH-POSIX) with the following permutations: * vanilla (outside of -mtune=generic) * With OpenBLAS * With OpenBLAS and -mtune=native * With OpenBLAS and -march=native * With OpenBLAS and link-time-optimization and -mtune=native * With OpenBLAS and link-time-optimization and -march=native Each time it passes make check-devel + make check-recommended (with the last, I haven't finished make check-devel as at work, I'm behind a firewall and the Internet2 test dies, preventing everything from segfault and on) I also build the 80-some packages I usually from source, and the only issues I've found so far is that 1) 'dplyr' doesn't compile well when specifying C++11 (either as c++111 or gnu++11) but compiles fine without it, 2) when using link-time-optimization, even though 'stringi' compiles from source, it causes a crash when 'knitr' is being installed. Installing 'stringi' as a binary (which is much faster than building it) obviates that problem, and 3) I'm having trouble with running 'rstan' (although it compiles as long as no C++11 option is passed) which I will bring to the attention of the rstan developers. I have not run the Rcpp segfault test as I always have problems with devtools, but you've said you're satisfied with that. Maybe over the weekend I can run some timing tests for comparison. Are there any tests in specific which would be useful? Avi