Rtools44
Thank you very much, Tomas, for your advice and your reminder about FLOSS. For completeness and closure, passing ""-Wa,-muse-unaligned-vector-move" to EOPTS in Mkrules.local allows make check-devel to complete with no errors. If there is a fund being set up to entice someone to write a good patch, please let me know off-line; while I don't think my company would contribute, I would contribute personal funds. Thanks again, Avi
On Tue, Mar 19, 2024 at 2:07?PM Tomas Kalibera <tomas.kalibera at gmail.com> wrote:
On 3/19/24 18:32, Avraham Adler wrote:
Thank you very much, Tomas. I guess when the variables were recast to long long, the compiler tried to use AVX2 commands.
This is just an optimization for quick copying of a C structure on the stack. This structure (regparams_t) has 8 ints, in this case.
This is rather disappointing from GCC's end; there isn't much we in R can do. As this bug has been around for over a dozen years, and the last comment is two years ago, I do not have hope that anything will be fixed in my lifetime. Being restricted to Windows, are the only two options to disable all AVX2 or use "-Wa,-muse-unaligned-vector-move"? There is no alternative easy-to-drop-in toolchain to build R from source on Windows, is there?
I would recommend you first check whether actually disabling AVX2 (or using "-Wa,-muse-unaligned-vector-move" with AVX2) makes any difference in performance for your use cases. Maybe not. There is no supported drop-in replacement toolchain that would work with R and packages on x86_64. It is very likely that some LLVM-based toolchain would work after a small amount of tweaks, as after all Rtools with LLVM is being used on aarch64, so R has been and packages are being fixed to work with it, but you would be on your own. And note that flang-new is not yet stable, so you might still have to use gfortran.
As someone who develops packages requiring source code but is restricted to Windows, I am loath to reduce the power of my code, unless I must. Any advice or suggestions you may have would be greatly appreciated.
GCC is like other open-source software. Submitting a good, well-tested patch, increases the chances things get fixed. Corporations that have the money can also pay experts specializing in compiler implementation to prepare such a patch. Tomas
Thank you for figuring this out, Avi On Tue, Mar 19, 2024 at 12:15?PM Tomas Kalibera <tomas.kalibera at gmail.com> wrote:
On 3/19/24 17:05, Tomas Kalibera wrote:
On 3/19/24 15:33, Avraham Adler wrote:
For completeness, I compiled R from source using the native BLAS, and make check fails again, this time on "aregexec". Running "agrep" from within Rgui causes it to crash to desktop. Could it be something to do with this: https://github.com/wch/r-source/commit/ba486d898c2698c2afb678abdab807510327541e? But that didn't affect aregexc or adist? This is beyond my understanding, but I'm happy to do any tests you need to help identify what is going on.
I can reproduce the problem (without LTO, without openblas), just by
"-O3 -march=native" and running
agrep("lasy", "1 lazy 2")
which crashes R.
The crash happens in do_agrep from agrep.c, when calling tre_regaexec:
} else {
const char *s = translateChar(STRING_ELT(vec, i));
if(mbcslocale && !mbcsValid(s))
error(_("input string %lld is invalid in this locale"),
(long long)i+1);
rc = tre_regaexec(®, s, &match, params, 0); // <===
here (line 242)
vmaxset(vmax);
}
0x00007fff79c642da <+1290>: mov 0x50(%rsp),%rdx
0x00007fff79c642df <+1295>: test %eax,%eax
0x00007fff79c642e1 <+1297>: je 0x7fff79c6522a <do_agrep+5210>
0x00007fff79c642e7 <+1303>: movl $0x0,0x20(%rsp)
0x00007fff79c642ef <+1311>: mov 0x38(%rsp),%rcx
0x00007fff79c642f4 <+1316>: mov %r15,%r8
0x00007fff79c642f7 <+1319>: lea 0x80(%rsp),%r9
0x00007fff79c642ff <+1327>: vmovdqu 0xb0(%rsp),%ymm4
=> 0x00007fff79c64308 <+1336>: vmovdqa %ymm4,0x80(%rsp)
0x00007fff79c64311 <+1345>: vzeroupper
0x00007fff79c64314 <+1348>: call 0x7fff79d183d0 <tre_regaexec>
And it happens because of incorrect alignment in an AVX2 instruction.
In the above, vmovdqa is used as a second step of copying the
regparams_t structure to the stack (to be passed as 4th argument of
tre_regaexec). But GCC uses 0x80(%rsp) for that, which happens only to
be 16-byte aligned, but not 32-byte aligned. AVX2 requires this to be
32-byte aligned in vmovdqa instruction, and hence the program segfaults.
This is a long standing bug in GCC that has been reported long time
ago: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=54412
It must be a coincidence that you haven't run into this with GCC 12.3
and earlier (so earlier versions of Rtools for Windows).
I think the simplest and most reliable work-around is to not use AVX2
with GCC on Windows.
You can also use "-Wa,-muse-unaligned-vector-move" option for the GNU assembler (add to optimization options), which will make the assembler use instructions that don't require memory to be aligned. Whether it would still be beneficial for performance to use AVX2 in the end or not probably depends on the code. Best Tomas
Best Tomas
Thank you, Avi On Mon, Mar 18, 2024 at 11:58?PM Avraham Adler <avraham.adler at gmail.com> wrote:
I ran "tools::testInstalledBasic("both")" from within Rterm, and it
failed on reg-tests-1a, and doesn't that last call in the FAIL look
familiar? Three times isn;t coincidence. What could be causing
agrep/adist to fail?
Avi
a <- c("NA", NA, "BANANA")
na <- NA_character_
a1 <- substr(a,1,1)
stopifnot(is.na(a1)==is.na(a))
a2 <- substring(a,1,1)
stopifnot(is.na(a2)==is.na(a))
a3 <- sub("NA","na",a)
stopifnot(is.na(a3)==is.na(a))
a3 <- gsub("NA","na",a)
stopifnot(is.na(a3)==is.na(a))
substr(a3, 1, 2) <- "na"
stopifnot(is.na(a3)==is.na(a))
substr(a3, 1, 2) <- na
stopifnot(all(is.na(a3)))
stopifnot(agrep("NA", a) == c(1, 3))
On Mon, Mar 18, 2024 at 11:52?PM Avraham Adler <avraham.adler at gmail.com> wrote:
Hello, Tomas. I ran check again and this time it failed on base. Base failed at "agrep" and utils failed at "adist" if that triggers any ideas. Thank you, Avi On Mon, Mar 18, 2024 at 11:49?PM Avraham Adler <avraham.adler at gmail.com> wrote:
Hello, Tomas.
I had hoped that my problem was that I was linking to OpenBLAS built
under Rtools43, but sadly that is not the case, as I built OpenBLAS
using Rtools44 and I still get the error in 'utils'. I do have
Rtools43 installed on this machine, but I am doing eveything from the
Rtools44 bash. My MkRules.local follows:
USE_ATLAS = YES
ATLAS_PATH = C:/R/OPB/OPB_03.26_1T_44
EOPTS = -march=native -pipe -mno-rtm
LTO = -flto=1 -fuse-linker-plugin
LTO_OPT = -flto=1 -fuse-linker-plugin
LTO_FC = -flto=1 -fuse-linker-plugin
LTO_FC_OPT = -flto=1 -fuse-linker-plugin
QPDF = C:/R/qpdf-11.9.0-msvc64
OPENMP = -fopenmp
I also make the following change to Makefile.win in
/src/extra/blas as
I have been doing for more than a decade:
--- /c/r/trunk/src/extra/blas/Makefile.win 2024-01-24
18:34:42.755255900 +0000
+++ /c/r/Makefile.win 2024-01-24 18:39:39.716458000 +0000
@@ -12,7 +12,7 @@
../../../$(BINDIR)/Rblas.dll: blas00.o ../../gnuwin32/dllversion.o
@$(ECHO) -------- Building $@ --------
$(DLL) -s -shared $(DLLFLAGS) -o $@ $^ Rblas.def \
- -L../../../$(IMPDIR) -lR -L"$(ATLAS_PATH)" -lf77blas -latlas
+ -L../../../$(IMPDIR) -lR -L"$(ATLAS_PATH)" -fopenmp
-lopenblas
else
../../../$(BINDIR)/Rblas.dll: blas.o blas2.o cmplxblas.o
cmplxblas2.o
../../gnuwin32/dllversion.o
@$(ECHO) -------- Building $@ --------
The utils-Ex.Rout.fail is 1188 lines long and I don't see any obvious
point of failure. The last few lines are:
cleanEx()
nameEx("adist")
### * adist
flush(stderr()); flush(stdout())
### Name: adist
### Title: Approximate String Distances
### Aliases: adist
### Keywords: character
### ** Examples
## Cf. https://en.wikipedia.org/wiki/Levenshtein_distance
adist("kitten", "sitting")
[,1] [1,] 3
## To see the transformation counts for the Levenshtein distance:
drop(attr(adist("kitten", "sitting", counts = TRUE), "counts"))
ins del sub
1 0 2
## To see the transformation sequences:
attr(adist(c("kitten", "sitting"), counts = TRUE), "trafos")
[,1] [,2] [1,] "MMMMMM" "SMMMSMI" [2,] "SMMMSMD" "MMMMMMM"
## Cf. the examples for agrep:
adist("lasy", "1 lazy 2")
[,1] [1,] 5
## For a "partial approximate match" (as used for agrep):
adist("lasy", "1 lazy 2", partial = TRUE)
The build works under Rtools43. Should I uninstall both versions of Rtools, my current R installation, and its library and try again? I'm doubtful that will help as my "active" R installation is in a completely different directory, but I am willing to try. If there is any output or other tests I can do, please let me know. Or, if you think I should raise this on r-devel. Thank you, Avi On Mon, Mar 18, 2024 at 4:08?AM Tomas Kalibera <tomas.kalibera at gmail.com> wrote:
Hello Avi, On 3/17/24 17:53, Avraham Adler wrote:
Hello, Tomas. As always, thank you for your incessant hard work. I have compiled R-devel 86144 using Rtools44 and it completes normally. However, it fails very early in make check-devel. Specifically, I get the output below, and have received it more than once. Is this something to raise on R-devel or is it an Rtools44-specific issue?
I can't tell from this output. Could you please try to get a more specific error output? Could you please share your compiler options (e.g. MkRules.local) and indeed if you made any modifications to the code? Also, as always, it is worth making sure that all code has been rebuilt using the new Rtools - e.g. delete any old package library for R-devel (4.4) you may have on your system. The testing done by myself and CRAN only covers the default compilation options as specified in the make files. Best Tomas
Thank you,
Avi
$ make check-devel
Testing examples for package 'base'
Testing examples for package 'tools'
comparing 'tools-Ex.Rout' to 'tools-Ex.Rout.save' ... NOTE
1046,1047d1045
< Warning in file(con, "r") :
< file("") only supports open = "w+" and open = "w+b":
using the former
1050,1051c1048,1049
< $ file : chr ""
< $ title : chr ""
---
> $ file : chr "grid.Rnw"
> $ title : chr "Introduction to grid"
Testing examples for package 'utils' Error: testing 'utils' failed Execution halted make[3]: *** [Makefile.win:29: test-Examples-Base] Error 1 make[2]: *** [Makefile.common:208: test-Examples] Error 2 make[1]: *** [Makefile.common:193: test-all-basics] Error 1 make: *** [Makefile:333: check-devel] Error 2 On Thu, Mar 14, 2024 at 4:45?AM Tomas Kalibera <tomas.kalibera at gmail.com> wrote:
Dear R Windows developers, there is now a new toolchain for R for Windows, Rtools44. It is now used by R-devel and is intended for R 4.4.0. Compared to Rtools43, it uses GCC 13 and updates other core components. See https://cran.r-project.org/bin/windows/Rtools/rtools44/news.html for a detailed list of changes. All users of R-devel who need to compile R packages with source code in C, C++ or Fortran should install Rtools44. From the user perspective, Rtools44 works the same way as Rtools43. See https://cran.r-project.org/bin/windows/base/howto-R-devel.html for instructions on how to build R-devel and packages for this version of R. It is recommended to re-install packages that need compilation to avoid potential incompatibilities with code built using Rtools43. Rtools44 also includes an experimental version for 64-bit ARM machines, using LLVM 17 (and clang, flang-new, lld, libc++) - the aarch64 version of Rtools has its own installer and distribution tarballs. Best Tomas
_______________________________________________ R-SIG-windows mailing list R-SIG-windows at r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-windows