OpenMP on CRAN

3 messages · Patrick Schratz, Simon Urbanek

Fri, Apr 3, 2020 5:33 AM #

Simon,

thanks. I assume that you mean ?clang from homebrew? = llvm (from homebrew)?
Indeed I am currently trying that out and it looks really robust for source installations (more than the systems clang (+ eventual openMP flags like suggested by Kevin) and other variants (gcc from homebrew or openMP enabled clang7 or 8).

Note that I do not use R via homebrew.

However, all in all I am essentially mixing the custom llvm from homebrew with the official R installer and the old 10.13 SDK.
It looks quite complex and I?d wish everything would be easier. However, in the end I just want to have a stable ?install from source? environment that works for all packages out there (I do not use the binaries).

All in all I am a bit confused now about all the mixing and options available. Let?s see what the foreseeable future brings and where things are going.

Regarding the SDK issue: I think most users just use the binaries on macOS (across all OS versions) and rarely face such issues. And there are presumably not many people out there that do actual R-devel testing on Catalina (?), otherwise I?d expect way more people to jump into the discussion. However, I am not alone with these issues as you can see in the Rcpp issue we were talking about.
Maybe you can even reproduce the issues by linking a custom install of the 10.15 SDK on a non-Catalina machine? I don?t know how much trouble that would bring up when trying to jump into the future - but this ist just an idea :)

Thanks for your work!

On 3. Apr 2020, 14:13 +0200, Simon Urbanek <simon.urbanek at r-project.org>, wrote:

Patrick,

you were commenting on the thread where we talked about CRAN R - that one is now compiled using Apple clang. You were talking about using clang from Homebrew - those are incompatible as they use different run-time. Unfortunately, the Intel OpenMP run-time varies by clang compiler version and is known to fail when used with the wrong compiler. Analogously, mixing gomp (from gcc) and iomp is problematic (and GNU Fortran uses gomp). As discussed in the Homebrew thread, if you are compiling everything via Homebrew including R then it's all good, you just can't mix Apple tools and Homebrew in general.

Also at the bottom I was only talking about 10.14 SDK - that's what we use on CRAN and I have not seen any issues with it - you were claiming that it doesn't work with Rcpp and igraph.

Cheers,
Simon

On 4/04/2020, at 1:00 AM, Patrick Schratz <patrick.schratz at gmail.com> wrote:

Simon,

I don?t understand fully. I am using llvm for all C variants (just now shown) in combination with the 10.13 SDK.
So far this combination works flawlessly for all ?problematic? packages like data.table, igraph or Rcpp.

I don?t have deeper knowledge about the ?iomp? setup but as of right now I don?t know what I am mixing up here. Can you shine more light on this?
If it is about llvm in general: data.table recommends to use llvm openly in their wiki due to the lack of openMP for the standard clang.
And while this could be handled as ?any other package? in R, we all know that it has quite some impact and probably devs who know what they do in terms of C configs.

Regarding the SDK issue: Both Rcpp v1.0.4 and igraph 1.2.5 can?t be installed from source with SDK 10.15 and standard apple clang (or any C compiler). Switching to SDK 10.13 solves the issue. You were even commenting on the Rcpp issue.
Since SDK 10.15 is the current release and many users have no choice on which OS version they are (for different reasons), this issue should imo be inspected in more detail. What do you think?

Best, Patrick
On 2. Apr 2020, 23:38 +0200, Simon Urbanek <simon.urbanek at r-project.org>, wrote:

Patrick,

you can't mix compilers - it really matters which iomp your'e using. llvm has its own modified version which may not be the same as the official Intel release. From your tests it looks like you're using the llvm one which will likely work only with that compiler. Since Apple doesn't have official omp support it's hard to say which versions work and which don't.

On 3/04/2020, at 10:20 AM, Patrick Schratz <patrick.schratz at gmail.com> wrote:

Thanks Kevin, interesting approach.

However, when testing it against a few packages I get symbol pointer issues (e.g. for data.table and xgboost).
Using llvm everything works.
Could llvm be the best middle way? Easy installation via brew and still clang compliant.

Currently my Makevars look as follows

CFLAGS=-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk
[?]

CC = ccache /usr/local/opt/llvm/bin/clang
[?]

Where llvm was installed via `brew install llvm`.
SDK 10.13 because of {igraph} and {Rcpp} issues with SDK 10.14 and SDK 10.15


I mentioned that before, but I do not see issues with 10.14 SDK.

Cheers,
Simon

On 2. Apr 2020, 23:01 +0200, Simon Urbanek <simon.urbanek at r-project.org>, wrote:

Tim,

On 3/04/2020, at 2:01 AM, BATES Timothy <tim.bates at ed.ac.uk> wrote:

Moving to a compiler that drops support for OpenMP seems a sad choice, especially now we?ve all climbed the learning curve of the non-Apple compiler (the real barrier was lack of a pkg installer and that?s done now).

It has caused a lot of issues, it trips people on a daily basis which is just not worth it. Also with Apple's increasingly strict rules about what can be distributed we don't want to be in the business in maintaining a separate toolchain. There have been numerous issues with C++ exceptions so the fall out was much bigger than originally expected and it would only get worse.

Losing OpenMP for the CRAN version of OpenMx/umx (our SEM packages) would be a big loss for users (for whom the CRAN version now supports OpenMP giving them a 2-12x speed up). In general, R on Mac is made more viable by having OpenMP

Re Brian?s points, I?d say that the distribution problem is crucial: Packages not on CRAN have dramatically diminished accessibility/useage.

The idea is that if a package deems this critical, it can enable this for the users. As it is now, packages can still supply iomp and use it.

That said, I would open for discussion the ability to distribute iomp with the R binary, but it would not be supported by R directly, i.e., it would be on the package author to make sure that the way the package operates is compatible with that binary. Let me know what you think.

Second, a great range of compute-intensive problems are amenable to division amongst cores, including nearly all models that take more than a nominal amount of time to run: So simulations, CIs, bootstrapping, nearly everything in genetics all speeds up.

Yes, but OpenMP is rarely used for those tasks. In R OpenMP can be only used for very small subset of such tasks - which is why alternative approaches are much more common.

Cheers,
Simon

I?d say especially on desktop/laptop. The big advantage of multi blade systems requires snowfall-type solutions, but desktops profit automatically from their multi-core structure and don;?t have multiple processors (except graphics, which no-one seems to be exploiting on CRAN-style R), so OpenMP is their one trick. I?d hope not to lose it.

Best, t

On 2 Apr 2020, at 05:18, Prof Brian Ripley <ripley at stats.ox.ac.uk> wrote:

On 01/04/2020 22:02, Simon Urbanek wrote:

JJB,
1. correct, there was too much trouble in this. But please feel free to start a new thread about this here if you have strong opinions.

Also note that it is possible (and not hard) to install packages from source with an OpenMP-supporting compiler, and how to do so is in the R-admin manual. The problems come in distributing them.

The benefits of OpenMP are often overestimated, especially on desktop/laptop level hardware. But it is available for the small (tiny?) proportion of users who need it.

The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

_______________________________________________
R-SIG-Mac mailing list
R-SIG-Mac at r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-mac

_______________________________________________
R-SIG-Mac mailing list
R-SIG-Mac at r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-mac

Simon Urbanek

Fri, Apr 3, 2020 12:54 PM #

Patrick,

LLVM is the compiler infrastructure project, clang is the C/C++ compiler from that project. We don't use any of the other compilers form LLVM.

Why don't use use Homebrew? That's exactly what Homebrew provides - its own ecosystem, everything form source (with cached binaries if you want them).

Mixing is not available. You pick one system and go with it, but can't mix between them, because of the different run-time libraries.

That's an entirely different, unrelated topic. Testing cutting edge (or even pre-release) systems makes sense, but that's not our current worry (there was a separate thread about CI).

Cheers,
Simon

Thanks for your work!
On 3. Apr 2020, 14:13 +0200, Simon Urbanek <simon.urbanek at r-project.org>, wrote:

Patrick,

you were commenting on the thread where we talked about CRAN R - that one is now compiled using Apple clang. You were talking about using clang from Homebrew - those are incompatible as they use different run-time. Unfortunately, the Intel OpenMP run-time varies by clang compiler version and is known to fail when used with the wrong compiler. Analogously, mixing gomp (from gcc) and iomp is problematic (and GNU Fortran uses gomp). As discussed in the Homebrew thread, if you are compiling everything via Homebrew including R then it's all good, you just can't mix Apple tools and Homebrew in general.

Also at the bottom I was only talking about 10.14 SDK - that's what we use on CRAN and I have not seen any issues with it - you were claiming that it doesn't work with Rcpp and igraph.

Cheers,
Simon

On 4/04/2020, at 1:00 AM, Patrick Schratz <patrick.schratz at gmail.com> wrote:

Simon,

I don?t understand fully. I am using llvm for all C variants (just now shown) in combination with the 10.13 SDK.
So far this combination works flawlessly for all ?problematic? packages like data.table, igraph or Rcpp.

I don?t have deeper knowledge about the ?iomp? setup but as of right now I don?t know what I am mixing up here. Can you shine more light on this?
If it is about llvm in general: data.table recommends to use llvm openly in their wiki due to the lack of openMP for the standard clang.
And while this could be handled as ?any other package? in R, we all know that it has quite some impact and probably devs who know what they do in terms of C configs.

Regarding the SDK issue: Both Rcpp v1.0.4 and igraph 1.2.5 can?t be installed from source with SDK 10.15 and standard apple clang (or any C compiler). Switching to SDK 10.13 solves the issue. You were even commenting on the Rcpp issue.
Since SDK 10.15 is the current release and many users have no choice on which OS version they are (for different reasons), this issue should imo be inspected in more detail. What do you think?

Best, Patrick
On 2. Apr 2020, 23:38 +0200, Simon Urbanek <simon.urbanek at r-project.org>, wrote:

Patrick,

you can't mix compilers - it really matters which iomp your'e using. llvm has its own modified version which may not be the same as the official Intel release. From your tests it looks like you're using the llvm one which will likely work only with that compiler. Since Apple doesn't have official omp support it's hard to say which versions work and which don't.

On 3/04/2020, at 10:20 AM, Patrick Schratz <patrick.schratz at gmail.com> wrote:

Thanks Kevin, interesting approach.

However, when testing it against a few packages I get symbol pointer issues (e.g. for data.table and xgboost).
Using llvm everything works.
Could llvm be the best middle way? Easy installation via brew and still clang compliant.

Currently my Makevars look as follows

CFLAGS=-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk
[?]

CC = ccache /usr/local/opt/llvm/bin/clang
[?]

Where llvm was installed via `brew install llvm`.
SDK 10.13 because of {igraph} and {Rcpp} issues with SDK 10.14 and SDK 10.15


I mentioned that before, but I do not see issues with 10.14 SDK.

Cheers,
Simon

On 2. Apr 2020, 23:01 +0200, Simon Urbanek <simon.urbanek at r-project.org>, wrote:

Tim,

On 3/04/2020, at 2:01 AM, BATES Timothy <tim.bates at ed.ac.uk> wrote:

Moving to a compiler that drops support for OpenMP seems a sad choice, especially now we?ve all climbed the learning curve of the non-Apple compiler (the real barrier was lack of a pkg installer and that?s done now).

It has caused a lot of issues, it trips people on a daily basis which is just not worth it. Also with Apple's increasingly strict rules about what can be distributed we don't want to be in the business in maintaining a separate toolchain. There have been numerous issues with C++ exceptions so the fall out was much bigger than originally expected and it would only get worse.

Losing OpenMP for the CRAN version of OpenMx/umx (our SEM packages) would be a big loss for users (for whom the CRAN version now supports OpenMP giving them a 2-12x speed up). In general, R on Mac is made more viable by having OpenMP

Re Brian?s points, I?d say that the distribution problem is crucial: Packages not on CRAN have dramatically diminished accessibility/useage.

The idea is that if a package deems this critical, it can enable this for the users. As it is now, packages can still supply iomp and use it.

That said, I would open for discussion the ability to distribute iomp with the R binary, but it would not be supported by R directly, i.e., it would be on the package author to make sure that the way the package operates is compatible with that binary. Let me know what you think.

Second, a great range of compute-intensive problems are amenable to division amongst cores, including nearly all models that take more than a nominal amount of time to run: So simulations, CIs, bootstrapping, nearly everything in genetics all speeds up.

Yes, but OpenMP is rarely used for those tasks. In R OpenMP can be only used for very small subset of such tasks - which is why alternative approaches are much more common.

Cheers,
Simon

I?d say especially on desktop/laptop. The big advantage of multi blade systems requires snowfall-type solutions, but desktops profit automatically from their multi-core structure and don;?t have multiple processors (except graphics, which no-one seems to be exploiting on CRAN-style R), so OpenMP is their one trick. I?d hope not to lose it.

Best, t

On 2 Apr 2020, at 05:18, Prof Brian Ripley <ripley at stats.ox.ac.uk> wrote:

On 01/04/2020 22:02, Simon Urbanek wrote:

JJB,
1. correct, there was too much trouble in this. But please feel free to start a new thread about this here if you have strong opinions.

Also note that it is possible (and not hard) to install packages from source with an OpenMP-supporting compiler, and how to do so is in the R-admin manual. The problems come in distributing them.

The benefits of OpenMP are often overestimated, especially on desktop/laptop level hardware. But it is available for the small (tiny?) proportion of users who need it.

The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

_______________________________________________
R-SIG-Mac mailing list
R-SIG-Mac at r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-mac

_______________________________________________
R-SIG-Mac mailing list
R-SIG-Mac at r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-mac

Patrick Schratz

Sat, Apr 4, 2020 12:34 AM #

Simon,

thanks. While this feels like a somewhat 1/1 discussion now, I also think that others might profit from it.

1. I am aware that LLVM is a wrapper bundling more than just the compiler and that it comes with its own copy of clang
2. I don?t use homebrew because it does not easily allow multiple R installations next to each other. This is possible via the official installers. Combining this with RSwitch from Bob I can switch between R interpreters with a keystroke. I am using homebrew for everything else and even contribute to some formulas from time to time.
3. In the end everyone if responsible for their own setup and deviating from standards make its more complicated to get help (if at all). However, I am still considering blogging about my setup in the future once I have found a robust one that worked for some time across all instances I faced issues with in the past (focusing on source installs). And don?t worry, I will make sure to point out that this is custom, what the CRAN setup is and the differences between homebrew and the CRAN thing. There are quite some myths floating around that this should be clarified in more depth on a static page (maybe even from you as the expert? ;) )

Regarding CI and testing SDK 10.15: A pointer to this discussion where the toolchain for GitHub Actions is discussed. I?ll move my thoughts about this to a new thread.

On 3. Apr 2020, 22:11 +0200, Simon Urbanek <simon.urbanek at r-project.org>, wrote:

Patrick,

On 4/04/2020, at 1:33 AM, Patrick Schratz <patrick.schratz at gmail.com> wrote:

Simon,

thanks. I assume that you mean ?clang from homebrew? = llvm (from homebrew)?

LLVM is the compiler infrastructure project, clang is the C/C++ compiler from that project. We don't use any of the other compilers form LLVM.

Indeed I am currently trying that out and it looks really robust for source installations (more than the systems clang (+ eventual openMP flags like suggested by Kevin) and other variants (gcc from homebrew or openMP enabled clang7 or 8).

Note that I do not use R via homebrew.

However, all in all I am essentially mixing the custom llvm from homebrew with the official R installer and the old 10.13 SDK.
It looks quite complex and I?d wish everything would be easier. However, in the end I just want to have a stable ?install from source? environment that works for all packages out there (I do not use the binaries).

Why don't use use Homebrew? That's exactly what Homebrew provides - its own ecosystem, everything form source (with cached binaries if you want them).

All in all I am a bit confused now about all the mixing and options available. Let?s see what the foreseeable future brings and where things are going.

Mixing is not available. You pick one system and go with it, but can't mix between them, because of the different run-time libraries.

Regarding the SDK issue: I think most users just use the binaries on macOS (across all OS versions) and rarely face such issues. And there are presumably not many people out there that do actual R-devel testing on Catalina (?), otherwise I?d expect way more people to jump into the discussion. However, I am not alone with these issues as you can see in the Rcpp issue we were talking about.
Maybe you can even reproduce the issues by linking a custom install of the 10.15 SDK on a non-Catalina machine? I don?t know how much trouble that would bring up when trying to jump into the future - but this ist just an idea :)

That's an entirely different, unrelated topic. Testing cutting edge (or even pre-release) systems makes sense, but that's not our current worry (there was a separate thread about CI).

Cheers,
Simon

Thanks for your work!
On 3. Apr 2020, 14:13 +0200, Simon Urbanek <simon.urbanek at r-project.org>, wrote:

Patrick,

you were commenting on the thread where we talked about CRAN R - that one is now compiled using Apple clang. You were talking about using clang from Homebrew - those are incompatible as they use different run-time. Unfortunately, the Intel OpenMP run-time varies by clang compiler version and is known to fail when used with the wrong compiler. Analogously, mixing gomp (from gcc) and iomp is problematic (and GNU Fortran uses gomp). As discussed in the Homebrew thread, if you are compiling everything via Homebrew including R then it's all good, you just can't mix Apple tools and Homebrew in general.

Also at the bottom I was only talking about 10.14 SDK - that's what we use on CRAN and I have not seen any issues with it - you were claiming that it doesn't work with Rcpp and igraph.

Cheers,
Simon

On 4/04/2020, at 1:00 AM, Patrick Schratz <patrick.schratz at gmail.com> wrote:

Simon,

I don?t understand fully. I am using llvm for all C variants (just now shown) in combination with the 10.13 SDK.
So far this combination works flawlessly for all ?problematic? packages like data.table, igraph or Rcpp.

I don?t have deeper knowledge about the ?iomp? setup but as of right now I don?t know what I am mixing up here. Can you shine more light on this?
If it is about llvm in general: data.table recommends to use llvm openly in their wiki due to the lack of openMP for the standard clang.
And while this could be handled as ?any other package? in R, we all know that it has quite some impact and probably devs who know what they do in terms of C configs.

Regarding the SDK issue: Both Rcpp v1.0.4 and igraph 1.2.5 can?t be installed from source with SDK 10.15 and standard apple clang (or any C compiler). Switching to SDK 10.13 solves the issue. You were even commenting on the Rcpp issue.
Since SDK 10.15 is the current release and many users have no choice on which OS version they are (for different reasons), this issue should imo be inspected in more detail. What do you think?

Best, Patrick
On 2. Apr 2020, 23:38 +0200, Simon Urbanek <simon.urbanek at r-project.org>, wrote:

Patrick,

you can't mix compilers - it really matters which iomp your'e using. llvm has its own modified version which may not be the same as the official Intel release. From your tests it looks like you're using the llvm one which will likely work only with that compiler. Since Apple doesn't have official omp support it's hard to say which versions work and which don't.

On 3/04/2020, at 10:20 AM, Patrick Schratz <patrick.schratz at gmail.com> wrote:

Thanks Kevin, interesting approach.

However, when testing it against a few packages I get symbol pointer issues (e.g. for data.table and xgboost).
Using llvm everything works.
Could llvm be the best middle way? Easy installation via brew and still clang compliant.

Currently my Makevars look as follows

CFLAGS=-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk
[?]

CC = ccache /usr/local/opt/llvm/bin/clang
[?]

Where llvm was installed via `brew install llvm`.
SDK 10.13 because of {igraph} and {Rcpp} issues with SDK 10.14 and SDK 10.15


I mentioned that before, but I do not see issues with 10.14 SDK.

Cheers,
Simon

On 2. Apr 2020, 23:01 +0200, Simon Urbanek <simon.urbanek at r-project.org>, wrote:

Tim,

On 3/04/2020, at 2:01 AM, BATES Timothy <tim.bates at ed.ac.uk> wrote:

Moving to a compiler that drops support for OpenMP seems a sad choice, especially now we?ve all climbed the learning curve of the non-Apple compiler (the real barrier was lack of a pkg installer and that?s done now).

It has caused a lot of issues, it trips people on a daily basis which is just not worth it. Also with Apple's increasingly strict rules about what can be distributed we don't want to be in the business in maintaining a separate toolchain. There have been numerous issues with C++ exceptions so the fall out was much bigger than originally expected and it would only get worse.

Losing OpenMP for the CRAN version of OpenMx/umx (our SEM packages) would be a big loss for users (for whom the CRAN version now supports OpenMP giving them a 2-12x speed up). In general, R on Mac is made more viable by having OpenMP

Re Brian?s points, I?d say that the distribution problem is crucial: Packages not on CRAN have dramatically diminished accessibility/useage.

The idea is that if a package deems this critical, it can enable this for the users. As it is now, packages can still supply iomp and use it.

That said, I would open for discussion the ability to distribute iomp with the R binary, but it would not be supported by R directly, i.e., it would be on the package author to make sure that the way the package operates is compatible with that binary. Let me know what you think.

Second, a great range of compute-intensive problems are amenable to division amongst cores, including nearly all models that take more than a nominal amount of time to run: So simulations, CIs, bootstrapping, nearly everything in genetics all speeds up.

Yes, but OpenMP is rarely used for those tasks. In R OpenMP can be only used for very small subset of such tasks - which is why alternative approaches are much more common.

Cheers,
Simon

I?d say especially on desktop/laptop. The big advantage of multi blade systems requires snowfall-type solutions, but desktops profit automatically from their multi-core structure and don;?t have multiple processors (except graphics, which no-one seems to be exploiting on CRAN-style R), so OpenMP is their one trick. I?d hope not to lose it.

Best, t

On 2 Apr 2020, at 05:18, Prof Brian Ripley <ripley at stats.ox.ac.uk> wrote:

On 01/04/2020 22:02, Simon Urbanek wrote:

JJB,
1. correct, there was too much trouble in this. But please feel free to start a new thread about this here if you have strong opinions.

Also note that it is possible (and not hard) to install packages from source with an OpenMP-supporting compiler, and how to do so is in the R-admin manual. The problems come in distributing them.

The benefits of OpenMP are often overestimated, especially on desktop/laptop level hardware. But it is available for the small (tiny?) proportion of users who need it.

The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

_______________________________________________
R-SIG-Mac mailing list
R-SIG-Mac at r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-mac

_______________________________________________
R-SIG-Mac mailing list
R-SIG-Mac at r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-mac