Changing the BLAS from openblas on a F32 box

Dear list,

What is the recommended incantation on Fedora 32 to swap out the
openblas BLAS that the packaged (rpm) version of R-core installs for
ATLAS?

If anyone is interested, the issue Simon Wood reports with openblas
and *mgcv* is:

Issues:

** openblas 0.3.x x<7 is not thread safe if itself compiled for single thread
   use and then called from multiple threads (unlike the reference BLAS, say).
   0.2.20 appears to be OK. For 0.3.x x>6 make USE_THREAD=0 USE_LOCKING=1
   to make openblas ensures thread safety (currently unclear if USE_LOCKING
   will be default from 0.3.7).

Hi Gavin,

On Wed, 27 May 2020 at 01:15, Gavin Simpson <ucfagls at gmail.com> wrote:

Dear list,

What is the recommended incantation on Fedora 32 to swap out the
openblas BLAS that the packaged (rpm) version of R-core installs for
ATLAS?

I'm afraid there is no official mechanism in place to do that yet.
There was a proposal [1], but it was never pushed forward due to some
issues and lack of time. There's a simple hack you can do though.

Since recently, R in Fedora no longer uses openblas-Rblas and links
against system openblas, as you can see here:

$ ldd /usr/lib64/R/bin/exec/R | grep blas
        libopenblas.so.0 => /lib64/libopenblas.so.0 (0x00007f8fff849000)

So one simple trick to override that is something along these lines:

$ mkdir ~/blas
$ ln -s /lib64/atlas/libsatlas.so.3 ~/blas/libopenblas.so.0
$ LD_LIBRARY_PATH=~/blas ldd /usr/lib64/R/bin/exec/R | grep blas
        libopenblas.so.0 => /home/****/blas/libopenblas.so.0
(0x00007f78b3ea2000)

As you can see, now R points to the link in my home, which in turn
points to system's atlas. Now you can define an alias in your .bashrc
to always prepend that override to R and Rscript. If you remove the
link in your home, it will default to system's openblas. If you point
it to another BLAS implementation, it will pick it up.

Hope it helps.

[1] https://fedoraproject.org/w/index.php?title=PackagingDrafts:BLAS_LAPACK

If anyone is interested, the issue Simon Wood reports with openblas
and *mgcv* is:

Issues:

** openblas 0.3.x x<7 is not thread safe if itself compiled for single thread
   use and then called from multiple threads (unlike the reference BLAS, say).
   0.2.20 appears to be OK. For 0.3.x x>6 make USE_THREAD=0 USE_LOCKING=1
   to make openblas ensures thread safety (currently unclear if USE_LOCKING
   will be default from 0.3.7).

Does this mean that single-threaded openblas should be built with
USE_LOCKING=1 by default? Is there any upstream recommendation about
this?

--
I?aki ?car

Of course, even a simpler trick is to launch R as follows:

LD_PRELOAD=/lib64/atlas/libsatlas.so.3 R

and then the symbols in libsatlas take precedence over libopenblas. Or
a mix between both alternatives, i.e., setting

LD_PRELOAD=/path/to/some/link R

and then change that link to point to openblas, atlas... Whatever
suits you best.

I?aki

On Wed, 27 May 2020 at 11:00, I?aki Ucar <iucar at fedoraproject.org> wrote:

Hi Gavin,

On Wed, 27 May 2020 at 01:15, Gavin Simpson <ucfagls at gmail.com> wrote:

Dear list,

What is the recommended incantation on Fedora 32 to swap out the
openblas BLAS that the packaged (rpm) version of R-core installs for
ATLAS?

I'm afraid there is no official mechanism in place to do that yet.
There was a proposal [1], but it was never pushed forward due to some
issues and lack of time. There's a simple hack you can do though.

Since recently, R in Fedora no longer uses openblas-Rblas and links
against system openblas, as you can see here:

$ ldd /usr/lib64/R/bin/exec/R | grep blas
        libopenblas.so.0 => /lib64/libopenblas.so.0 (0x00007f8fff849000)

So one simple trick to override that is something along these lines:

$ mkdir ~/blas
$ ln -s /lib64/atlas/libsatlas.so.3 ~/blas/libopenblas.so.0
$ LD_LIBRARY_PATH=~/blas ldd /usr/lib64/R/bin/exec/R | grep blas
        libopenblas.so.0 => /home/****/blas/libopenblas.so.0
(0x00007f78b3ea2000)

As you can see, now R points to the link in my home, which in turn
points to system's atlas. Now you can define an alias in your .bashrc
to always prepend that override to R and Rscript. If you remove the
link in your home, it will default to system's openblas. If you point
it to another BLAS implementation, it will pick it up.

Hope it helps.

[1] https://fedoraproject.org/w/index.php?title=PackagingDrafts:BLAS_LAPACK

If anyone is interested, the issue Simon Wood reports with openblas
and *mgcv* is:

Issues:

** openblas 0.3.x x<7 is not thread safe if itself compiled for single thread
   use and then called from multiple threads (unlike the reference BLAS, say).
   0.2.20 appears to be OK. For 0.3.x x>6 make USE_THREAD=0 USE_LOCKING=1
   to make openblas ensures thread safety (currently unclear if USE_LOCKING
   will be default from 0.3.7).

Does this mean that single-threaded openblas should be built with
USE_LOCKING=1 by default? Is there any upstream recommendation about
this?

--
I?aki ?car

--
I?aki ?car

Thanks I?aki, that is exactly what i was looking for, esp the last
option which I have now configured as an alias for easy remembering.

I can answer the question re USE_LOCKING=1. I think that using both
those options is required to get thread-safety even if openblas was
compiled for single thread use. I don't know to what extent Simon has
engaged with upstream on this etc.

All I know is that using the openblas shipped with Fedora for R is
currently a recipe for disaster for the large GAMs we're trying to
fit. But being able to switch to atlas temporarily is a good
alternative.

On Wed, 27 May 2020 at 21:40, Gavin Simpson <ucfagls at gmail.com> wrote:

Thanks I?aki, that is exactly what i was looking for, esp the last
option which I have now configured as an alias for easy remembering.

I can answer the question re USE_LOCKING=1. I think that using both
those options is required to get thread-safety even if openblas was
compiled for single thread use. I don't know to what extent Simon has
engaged with upstream on this etc.

I've seen that there is a brief note about this in the project's wiki.
I agree that a sensible default in any distro would be to build
openblas-serial with USE_LOCKING=1. But I don't understand why there
is no recommendation upstream (or I didn't find it besides the note in
the wiki) and there's no idea about the performance penalty that we
incur doing so. I brought this topic to fedora-devel.

All I know is that using the openblas shipped with Fedora for R is
currently a recipe for disaster for the large GAMs we're trying to
fit. But being able to switch to atlas temporarily is a good
alternative.

Note that switching to openblas-openmp (libopenblaso.so) should be
thread-safe and will probably get you a better performance than Atlas.
Also, Fedora packages blis (which provides
/lib64/blisblas/libblas.so.3). It seems to be thread-safe should be
more performant than Atlas too.

--
I?aki ?car

Thanks (again) I?aki.

There was a typo in my reply above. I should have said: I *can't*
answer the question re USE_LOCKING=1.

Those other suggestions are really helpful too; I really didn't
understand what the difference was (I'm still not clear what the
differences are between say openblas-openmp and openblas-openmp64),
but I did get R to pass mgcv's thread safe test with both
openblas-openmp and blis-openmp, so I have aliased those options for
use too.

On Wed, 27 May 2020 at 23:03, Gavin Simpson <ucfagls at gmail.com> wrote:

Thanks (again) I?aki.

There was a typo in my reply above. I should have said: I *can't*
answer the question re USE_LOCKING=1.

:)

Those other suggestions are really helpful too; I really didn't
understand what the difference was (I'm still not clear what the
differences are between say openblas-openmp and openblas-openmp64),
but I did get R to pass mgcv's thread safe test with both
openblas-openmp and blis-openmp, so I have aliased those options for
use too.

Basically, openblas has a number of features that can be enabled or
disabled: 64-bit integer support, threading and parallelization of
certain parts using openmp (as, e.g., data.table does). With the
combination of these features, we end up with many different flavors,
and all these are compiled and sub-packaged separately.

Usually, you want parallelization at the top level calling a serial
version, as mgcv does. Otherwise, you may end up with an "explosion"
of threads that is counterproductive.

--
I?aki ?car