I've been going around in circles trying to get BLAS-switching working on a current r-devel, I'm sure I'm doing something dumb. Any ideas about what I might be doing wrong, or suggestions for further diagnosis, would be welcome! tl;dr I am compiling R-devel with (to the best of my knowledge) options set to allow BLAS-switching, but getting "undefined symbol" errors. ---- Latest R-devel (via SVN), PopOS!/Ubuntu 20.10 I have read Dirk E's post: https://github.com/eddelbuettel/mkl4deb I have attempted to read the relevant section of R Installation & Administration several times: https://cran.r-project.org/doc/manuals/r-release/R-admin.html#BLAS https://wiki.debian.org/DebianScience/LinearAlgebraLibraries I have installed MKL and OpenBLAS on my system via 'apt install' (libopenblas-dev, libopenblas-base, and TWO versions of intel-mkl-64bit) When I build R without BLAS everything is OK; rm -Rf r-build; mkdir r-build; cd r-build; ../r-devel/configure --without-blas --enable-R-shlib --enable-BLAS-shlib; make -j 6 Matrix products: default BLAS: /usr/local/lib/R/lib/libRblas.so LAPACK: /usr/local/lib/R/lib/libRlapack.so When I look at my BLAS alternatives I don't see anything obviously wrong: sudo update-alternatives --config libblas.so.3-x86_64-linux-gnu There are 3 choices for the alternative libblas.so.3-x86_64-linux-gnu (providing /usr/lib/x86_64-linux-gnu/libblas.so.3). Selection Path Priority Status ------------------------------------------------------------ * 0 /opt/intel/mkl/lib/intel64/libmkl_rt.so 150 auto mode 1 /opt/intel/mkl/lib/intel64/libmkl_rt.so 150 manual mode 2 /usr/lib/x86_64-linux-gnu/blas/libblas.so.3 10 manual mode 3 /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3 100 manual mode When I rebuild R with --with-blas: rm -Rf r-build; mkdir r-build; cd r-build; ../r-devel/configure --with-blas --enable-R-shlib --enable-BLAS-shlib; make -j 6 I end up with this: gcc -I../../../r-devel/src/extra -I/usr/include/tirpc -I. -I../../src/include -I../../../r-devel/src/include -I/usr/local/include -I../../../r-devel/src/nmath -DHAVE_CONFIG_H -fopenmp -fpic -g -O2 -c ../../../r-devel/src/main/Rmain.c -o Rmain.o gcc -Wl,--export-dynamic -fopenmp -L"../../lib" -L/usr/local/lib -o R.bin Rmain.o -lR -lRblas /usr/bin/ld: ../../lib/libR.so: undefined reference to `zgemm_' /usr/bin/ld: ../../lib/libR.so: undefined reference to `daxpy_' /usr/bin/ld: ../../lib/libR.so: undefined reference to `dgemv_' /usr/bin/ld: ../../lib/libR.so: undefined reference to `dscal_' If ======================= intel-mkl-64bit-2018.2-046/all,now 2018.2-046 amd64 [installed] intel-mkl-64bit-2020.4-912/all,now 2020.4-912 amd64 [installed] <... lots more intel-mkl stuff> libblas-dev/groovy,now 3.9.0-3ubuntu1 amd64 [installed,automatic] libblas3/groovy,now 3.9.0-3ubuntu1 amd64 [installed,automatic] libgraphblas3/groovy,now 1:5.8.1+dfsg-2 amd64 [installed,automatic] libgslcblas0/groovy,now 2.6+dfsg-2 amd64 [installed,automatic] libopenblas-base/groovy,now 0.3.10+ds-3ubuntu1 amd64 [installed] libopenblas-dev/groovy,now 0.3.10+ds-3ubuntu1 amd64 [installed] libopenblas-pthread-dev/groovy,now 0.3.10+ds-3ubuntu1 amd64 [installed,automatic] libopenblas0-pthread/groovy,now 0.3.10+ds-3ubuntu1 amd64 [installed,automatic] libopenblas0/groovy,now 0.3.10+ds-3ubuntu1 amd64 [installed]
boneheaded BLAS questions
10 messages · Ben Bolker, Dirk Eddelbuettel, Simon Urbanek +2 more
Ben, This stuff has worked unchanged since the 1990s when we had a _really_ far sighted fellow in Debian come up with the 'switch the links' scheme which was (and is) subsequently deployed by many numerical applications within Debian, R and e.g. Octave included. And I used this ability to switch over a decade ago in a never-quite-finished paper which resulted in a package as well as a vignette as paper draft on CRAN: gcbd [1] It used the ability to switch between implementation to time and compare and benchmark the various BLAS and LAPACK libraries -- which was then motivated by a comparison with GPUs. (The actual code / package is stale-ish as some of the underlying packages have gone as eg the GPU one -- but the mechanics you are after still work the exact same way on Debian and derivarives including Ubuntu and PopOS.) (As a complete aside, the state of the art here is now one level up in libraries based on flame/blis (a riff on blas) which can do a similar logical switch _at runtime_ (rather than by flipping softlinks and restarting the app). Julia and some other languages uses that, I think Fedora may have it in its R build as well. Inaki may know more...) That said, from the top of my head, I think you error may just be with the second R compilation -- I always (i.e. for the Debian package) use both --with-blas --with-lapack and not just --with-blas. And I do there is public: if you know where to look you can see the exact invocation of the Debian build of the R package (which Ubuntu and Pop and ... then shadow) [2] Hth, Dirk [1] https://cran.r-project.org/package=gcbd [2] https://sources.debian.org/src/r-base/4.0.4-1/debian/rules/ (and I apologise for how messy this still is)
https://dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org
Thanks. I know it's supposed to Just Work (and I definitely
appreciate all the work that's gone into making it Just Work 99% of the
time!).
I tried --with-lapack, no joy.
Will try to decipher the rules file tomorrow ...
cheers
Ben
On 3/17/21 10:25 PM, Dirk Eddelbuettel wrote:
Ben, This stuff has worked unchanged since the 1990s when we had a _really_ far sighted fellow in Debian come up with the 'switch the links' scheme which was (and is) subsequently deployed by many numerical applications within Debian, R and e.g. Octave included. And I used this ability to switch over a decade ago in a never-quite-finished paper which resulted in a package as well as a vignette as paper draft on CRAN: gcbd [1] It used the ability to switch between implementation to time and compare and benchmark the various BLAS and LAPACK libraries -- which was then motivated by a comparison with GPUs. (The actual code / package is stale-ish as some of the underlying packages have gone as eg the GPU one -- but the mechanics you are after still work the exact same way on Debian and derivarives including Ubuntu and PopOS.) (As a complete aside, the state of the art here is now one level up in libraries based on flame/blis (a riff on blas) which can do a similar logical switch _at runtime_ (rather than by flipping softlinks and restarting the app). Julia and some other languages uses that, I think Fedora may have it in its R build as well. Inaki may know more...) That said, from the top of my head, I think you error may just be with the second R compilation -- I always (i.e. for the Debian package) use both --with-blas --with-lapack and not just --with-blas. And I do there is public: if you know where to look you can see the exact invocation of the Debian build of the R package (which Ubuntu and Pop and ... then shadow) [2] Hth, Dirk [1] https://cran.r-project.org/package=gcbd [2] https://sources.debian.org/src/r-base/4.0.4-1/debian/rules/ (and I apologise for how messy this still is)
Ben, possibly useful project related to this https://github.com/staticfloat/libblastrampoline which is what Dirk referred to as the Julia state of art. It is actually much more complex than it sounds because of differences in naming and ABI between BLAS implementations, so simple switches don't work. Dirk has the luxury of having control over what he compiles, but that is not always the case (like with Accelerate or MKL) ... Cheers, Simon
On 18/03/2021, at 3:53 PM, Ben Bolker <bbolker at gmail.com> wrote: Thanks. I know it's supposed to Just Work (and I definitely appreciate all the work that's gone into making it Just Work 99% of the time!). I tried --with-lapack, no joy. Will try to decipher the rules file tomorrow ... cheers Ben On 3/17/21 10:25 PM, Dirk Eddelbuettel wrote:
Ben, This stuff has worked unchanged since the 1990s when we had a _really_ far sighted fellow in Debian come up with the 'switch the links' scheme which was (and is) subsequently deployed by many numerical applications within Debian, R and e.g. Octave included. And I used this ability to switch over a decade ago in a never-quite-finished paper which resulted in a package as well as a vignette as paper draft on CRAN: gcbd [1] It used the ability to switch between implementation to time and compare and benchmark the various BLAS and LAPACK libraries -- which was then motivated by a comparison with GPUs. (The actual code / package is stale-ish as some of the underlying packages have gone as eg the GPU one -- but the mechanics you are after still work the exact same way on Debian and derivarives including Ubuntu and PopOS.) (As a complete aside, the state of the art here is now one level up in libraries based on flame/blis (a riff on blas) which can do a similar logical switch _at runtime_ (rather than by flipping softlinks and restarting the app). Julia and some other languages uses that, I think Fedora may have it in its R build as well. Inaki may know more...) That said, from the top of my head, I think you error may just be with the second R compilation -- I always (i.e. for the Debian package) use both --with-blas --with-lapack and not just --with-blas. And I do there is public: if you know where to look you can see the exact invocation of the Debian build of the R package (which Ubuntu and Pop and ... then shadow) [2] Hth, Dirk [1] https://cran.r-project.org/package=gcbd [2] https://sources.debian.org/src/r-base/4.0.4-1/debian/rules/ (and I apologise for how messy this still is)
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
On 17 March 2021 at 22:53, Ben Bolker wrote:
| Thanks. I know it's supposed to Just Work (and I definitely | appreciate all the work that's gone into making it Just Work 99% of the | time!). And for what it is worth, the aforementioned 'switching from within' solution is using FlexiBLAS (not BLIS as I had said in the previous email), and was described in an R application R here: https://www.enchufa2.es/archives/switch-blas-lapack-without-leaving-your-r-session.html That won't help for you tried on your Debian-based system though, and I would (in the near-term) try that. Dirk
https://dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org
On 3/18/21 3:53 AM, Ben Bolker wrote:
Thanks.? I know it's supposed to Just Work (and I definitely appreciate all the work that's gone into making it Just Work 99% of the time!). ? I tried --with-lapack, no joy. ? Will try to decipher the rules file tomorrow ...
This is documented in R Admin manual, section A.3, and there is also "configure --help". On my Ubuntu 20.04, using "--with-blas --with-lapack" when a BLAS/LAPACK implementation is installed via "apt" works for me: with libblas3, liblapack3 I get in R via sessionInfo() BLAS:?? /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.9.0 LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0 then I install libopenblas0 and get, after re-starting R (not rebuilding): BLAS:?? /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3 LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/liblapack.so.3 Best Tomas
? cheers ?? Ben On 3/17/21 10:25 PM, Dirk Eddelbuettel wrote:
Ben, This stuff has worked unchanged since the 1990s when we had a _really_ far sighted fellow in Debian come up with the 'switch the links' scheme which was (and is) subsequently deployed by many numerical applications within Debian, R and e.g. Octave included. And I used this ability to switch over a decade ago in a never-quite-finished paper which resulted in a package as well as a vignette as paper draft on CRAN: gcbd [1] It used the ability to switch between implementation to time and compare and benchmark the various BLAS and LAPACK libraries -- which was then motivated by a comparison with GPUs. (The actual code / package is stale-ish as some of the underlying packages have gone as eg the GPU one -- but the mechanics you are after still work the exact same way on Debian and derivarives including Ubuntu and PopOS.) (As a complete aside, the state of the art here is now one level up in libraries based on flame/blis (a riff on blas) which can do a similar logical switch _at runtime_ (rather than by flipping softlinks and restarting the app). Julia and some other languages uses that, I think Fedora may have it in its R build as well. Inaki may know more...) That said, from the top of my head, I think you error may just be with the second R compilation -- I always (i.e. for the Debian package) use both ?? --with-blas --with-lapack and not just --with-blas. And I do there is public: if you know where to look you can see the exact invocation of the Debian build of the R package (which Ubuntu and Pop and ... then shadow) [2] Hth, Dirk [1] https://cran.r-project.org/package=gcbd [2] https://sources.debian.org/src/r-base/4.0.4-1/debian/rules/ ???? (and I apologise for how messy this still is)
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
On 3/18/21 9:15 AM, Tomas Kalibera wrote:
On 3/18/21 3:53 AM, Ben Bolker wrote:
Thanks.? I know it's supposed to Just Work (and I definitely appreciate all the work that's gone into making it Just Work 99% of the time!). ? I tried --with-lapack, no joy. ? Will try to decipher the rules file tomorrow ...
This is documented in R Admin manual, section A.3, and there is also "configure --help". On my Ubuntu 20.04, using "--with-blas --with-lapack" when a BLAS/LAPACK implementation is installed via "apt" works for me: with libblas3, liblapack3 I get in R via sessionInfo() BLAS:?? /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.9.0 LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0 then I install libopenblas0 and get, after re-starting R (not rebuilding): BLAS:?? /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3 LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/liblapack.so.3
and to install say MKL, this works for me: apt-get install intel-mkl-full and then: env MKL_INTERFACE_LAYER=GNU,LP64 MKL_THREADING_LAYER=GNU R gives me: BLAS/LAPACK: /usr/lib/x86_64-linux-gnu/libmkl_rt.so This is following documentation in A.3.1.3 of R Admin manual. Best Tomas
Best Tomas
? cheers ?? Ben On 3/17/21 10:25 PM, Dirk Eddelbuettel wrote:
Ben, This stuff has worked unchanged since the 1990s when we had a _really_ far sighted fellow in Debian come up with the 'switch the links' scheme which was (and is) subsequently deployed by many numerical applications within Debian, R and e.g. Octave included. And I used this ability to switch over a decade ago in a never-quite-finished paper which resulted in a package as well as a vignette as paper draft on CRAN: gcbd [1] It used the ability to switch between implementation to time and compare and benchmark the various BLAS and LAPACK libraries -- which was then motivated by a comparison with GPUs. (The actual code / package is stale-ish as some of the underlying packages have gone as eg the GPU one -- but the mechanics you are after still work the exact same way on Debian and derivarives including Ubuntu and PopOS.) (As a complete aside, the state of the art here is now one level up in libraries based on flame/blis (a riff on blas) which can do a similar logical switch _at runtime_ (rather than by flipping softlinks and restarting the app). Julia and some other languages uses that, I think Fedora may have it in its R build as well. Inaki may know more...) That said, from the top of my head, I think you error may just be with the second R compilation -- I always (i.e. for the Debian package) use both ?? --with-blas --with-lapack and not just --with-blas. And I do there is public: if you know where to look you can see the exact invocation of the Debian build of the R package (which Ubuntu and Pop and ... then shadow) [2] Hth, Dirk [1] https://cran.r-project.org/package=gcbd [2] https://sources.debian.org/src/r-base/4.0.4-1/debian/rules/ ???? (and I apologise for how messy this still is)
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
On Thu, 18 Mar 2021 at 05:10, Dirk Eddelbuettel <edd at debian.org> wrote:
On 17 March 2021 at 22:53, Ben Bolker wrote: | Thanks. I know it's supposed to Just Work (and I definitely | appreciate all the work that's gone into making it Just Work 99% of the | time!). And for what it is worth, the aforementioned 'switching from within' solution is using FlexiBLAS (not BLIS as I had said in the previous email), and was described in an R application R here: https://www.enchufa2.es/archives/switch-blas-lapack-without-leaving-your-r-session.html
Thanks, Dirk. Yes, since Fedora 33 (current release), we leverage this excellent work by Martin K?hler et al. [1], so that every BLAS/LAPACK consumer in Fedora is linked against FlexiBLAS, which enables transparent live switching. And there are R and octave packages providing bindings, as shown in the post above. Julia is in fact the only component that is currently *not* using it due to the particularities of their BLAS/LAPACK stack management, but they are interested in FlexiBLAS too and some work is underway [2]. If you are interested in this, Ben, you could compile FlexiBLAS yourself, docs are very clear and it's pretty straightforward. And then you only need to tell R to link against libflexiblas. For that, as previously described, we use (see lines 691-693 in [3]): --with-lapack --with-blas=flexiblas A small tweak in the configure is required though (see line 679 in [3]; in fact, I should port a proper fix upstream, but I didn't find the time yet). And if you have any issue Martin or myself could help. [1] https://www.mpi-magdeburg.mpg.de/projects/flexiblas [2] https://github.com/mpimd-csc/flexiblas/issues/12 [3] https://src.fedoraproject.org/rpms/R/blob/rawhide/f/R.spec
I?aki ?car
On 18 March 2021 at 09:15, Tomas Kalibera wrote:
| This is documented in R Admin manual, section A.3, and there is also | "configure --help". | | On my Ubuntu 20.04, using "--with-blas --with-lapack" when a BLAS/LAPACK | implementation is installed via "apt" works for me: | | with libblas3, liblapack3 I get in R via sessionInfo() | | BLAS:?? /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.9.0 | LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0 | | then I install libopenblas0 and get, after re-starting R (not rebuilding): | | BLAS:?? /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3 | LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/liblapack.so.3 Yes, thank you, that is exactly what I use and recommend too. And ...
On 18 March 2021 at 09:34, Tomas Kalibera wrote:
| and to install say MKL, this works for me: | | apt-get install intel-mkl-full | | and then: | | env MKL_INTERFACE_LAYER=GNU,LP64 MKL_THREADING_LAYER=GNU R | | gives me: | | BLAS/LAPACK: /usr/lib/x86_64-linux-gnu/libmkl_rt.so | | This is following documentation in A.3.1.3 of R Admin manual. ... which is what the (surprisingly popular, 139 stars) simple two-year old script at GitHub has been automated. https://github.com/eddelbuettel/mkl4deb I may need to update the recommendation for the two MKL_* variables. Dirk
https://dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org
For what it's worth I eventually got it to build in a hacky way (had
to add -lopenblaslib manually). FWIW I *did* RTFM, several times, but
for whatever reason the standard recipes are not working for me ...
thanks!
Ben Bolker
On 3/18/21 7:52 AM, Dirk Eddelbuettel wrote:
On 18 March 2021 at 09:15, Tomas Kalibera wrote:
| This is documented in R Admin manual, section A.3, and there is also
| "configure --help".
|
| On my Ubuntu 20.04, using "--with-blas --with-lapack" when a BLAS/LAPACK
| implementation is installed via "apt" works for me:
|
| with libblas3, liblapack3 I get in R via sessionInfo()
|
| BLAS:?? /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.9.0
| LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0
|
| then I install libopenblas0 and get, after re-starting R (not rebuilding):
|
| BLAS:?? /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
| LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/liblapack.so.3
Yes, thank you, that is exactly what I use and recommend too. And ...
On 18 March 2021 at 09:34, Tomas Kalibera wrote:
| and to install say MKL, this works for me:
|
| apt-get install intel-mkl-full
|
| and then:
|
| env MKL_INTERFACE_LAYER=GNU,LP64 MKL_THREADING_LAYER=GNU R
|
| gives me:
|
| BLAS/LAPACK: /usr/lib/x86_64-linux-gnu/libmkl_rt.so
|
| This is following documentation in A.3.1.3 of R Admin manual.
... which is what the (surprisingly popular, 139 stars) simple two-year old
script at GitHub has been automated.
https://github.com/eddelbuettel/mkl4deb
I may need to update the recommendation for the two MKL_* variables.
Dirk