Skip to content

Getting multithreaded R working (solution)

9 messages · Stefan Theussl, Dirk Eddelbuettel, Jonathan Greenberg +1 more

Original
Stefan Theussl
# 
[...]
 > P.S. If anyone has *easy* instructions for getting atlas working as a
 > multithreaded library or how do drop in the Intel MKL (I couldn't
 > figure out which *.so would be the correct one) please let me know and
 > I'll post them as well.

I did get MKL working with R via the following trick:

1) export 
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/intel/intel-mkl-9.0.018/lib/em64t

2) then configure with: 
"--with-blas=-L/opt/intel/intel-mkl-9.0.018/lib/em64t/ -lmkl -lguide 
-lpthread"

Back then I used MKL version 9.0.018. I don't know if this still works 
with more recent MKL or R.

Best,
st
Jonathan Greenberg
# 
Thanks Stefan:

Have you (or anyone) gotten Intel MKL working via the "plugin"
approach that Dirk has setup for the debian packages, where you just
point:

update-alternatives --config libblas.so.3gf
and
update-alternatives --config liblapack.so
to the BLAS/lapack libraries you want --  I don't know which file to
point to for this.

ls -l /usr/lib/intel/mkl/10.2.5.035/lib/em64t returned (a standard
install points to /opt/intel by the way, I moved these):

-r-xr-xr-x 1 root root    984748 Mar 25 09:05 libguide.a
-r-xr-xr-x 1 root root    698155 Mar 25 09:05 libguide.so
-r-xr-xr-x 1 root root   1056924 Mar 25 09:05 libiomp5.a
-r-xr-xr-x 1 root root    749714 Mar 25 09:05 libiomp5.so
-r-xr-xr-x 1 root root  19930316 Mar 25 12:04 libmkl_avx.so
-r--r--r-- 1 root root   1592136 Mar 25 12:10 libmkl_blacs_ilp64.a
-r--r--r-- 1 root root   1593520 Mar 25 12:09 libmkl_blacs_intelmpi_ilp64.a
-r-xr-xr-x 1 root root    639316 Mar 25 12:09 libmkl_blacs_intelmpi_ilp64.so
-r--r--r-- 1 root root    973850 Mar 25 12:05 libmkl_blacs_intelmpi_lp64.a
-r-xr-xr-x 1 root root    404221 Mar 25 12:05 libmkl_blacs_intelmpi_lp64.so
-r--r--r-- 1 root root    972474 Mar 25 12:06 libmkl_blacs_lp64.a
-r--r--r-- 1 root root   1620802 Mar 25 12:12 libmkl_blacs_openmpi_ilp64.a
-r--r--r-- 1 root root   1000990 Mar 25 12:07 libmkl_blacs_openmpi_lp64.a
-r--r--r-- 1 root root   1593152 Mar 25 12:13 libmkl_blacs_sgimpt_ilp64.a
-r--r--r-- 1 root root    973498 Mar 25 12:08 libmkl_blacs_sgimpt_lp64.a
-r--r--r-- 1 root root    447246 Mar 25 12:36 libmkl_blas95_ilp64.a
-r--r--r-- 1 root root    435758 Mar 25 12:36 libmkl_blas95_lp64.a
-r--r--r-- 1 root root    112558 Mar 25 12:13 libmkl_cdft_core.a
-r--r--r-- 1 root root 210211710 Mar 25 12:39 libmkl_core.a
-r-xr-xr-x 1 root root   3888898 Mar 25 12:03 libmkl_core.so
-r-xr-xr-x 1 root root  19385503 Mar 25 12:03 libmkl_def.so
-r--r--r-- 1 root root  10037880 Mar 25 12:38 libmkl_gf_ilp64.a
-r-xr-xr-x 1 root root   3583798 Mar 25 12:39 libmkl_gf_ilp64.so
-r--r--r-- 1 root root  10491422 Mar 25 12:38 libmkl_gf_lp64.a
-r-xr-xr-x 1 root root   3899087 Mar 25 12:39 libmkl_gf_lp64.so
-r--r--r-- 1 root root  15441782 Mar 25 12:39 libmkl_gnu_thread.a
-r-xr-xr-x 1 root root  12457443 Mar 25 12:40 libmkl_gnu_thread.so
-r--r--r-- 1 root root  10363194 Mar 25 12:38 libmkl_intel_ilp64.a
-r-xr-xr-x 1 root root   3719554 Mar 25 12:39 libmkl_intel_ilp64.so
-r--r--r-- 1 root root  10816706 Mar 25 12:38 libmkl_intel_lp64.a
-r-xr-xr-x 1 root root   4034675 Mar 25 12:39 libmkl_intel_lp64.so
-r--r--r-- 1 root root   2537716 Mar 25 11:55 libmkl_intel_sp2dp.a
-r-xr-xr-x 1 root root   1175095 Mar 25 12:04 libmkl_intel_sp2dp.so
-r--r--r-- 1 root root  26877982 Mar 25 12:39 libmkl_intel_thread.a
-r-xr-xr-x 1 root root  20805097 Mar 25 12:39 libmkl_intel_thread.so
-r--r--r-- 1 root root   6701900 Mar 25 12:36 libmkl_lapack95_ilp64.a
-r--r--r-- 1 root root   6693188 Mar 25 12:36 libmkl_lapack95_lp64.a
-r-xr-xr-x 1 root root  13269122 Mar 25 12:04 libmkl_lapack.so
-r-xr-xr-x 1 root root  22193012 Mar 25 12:03 libmkl_mc3.so
-r-xr-xr-x 1 root root  22455155 Mar 25 12:03 libmkl_mc.so
-r-xr-xr-x 1 root root  20821233 Mar 25 12:03 libmkl_p4n.so
-r--r--r-- 1 root root  18304180 Mar 25 12:39 libmkl_pgi_thread.a
-r-xr-xr-x 1 root root  15045661 Mar 25 12:40 libmkl_pgi_thread.so
-r--r--r-- 1 root root   9924200 Mar 25 12:40 libmkl_scalapack_ilp64.a
-r-xr-xr-x 1 root root   6208757 Mar 25 12:40 libmkl_scalapack_ilp64.so
-r--r--r-- 1 root root   9882058 Mar 25 12:40 libmkl_scalapack_lp64.a
-r-xr-xr-x 1 root root   6123754 Mar 25 12:40 libmkl_scalapack_lp64.so
-r--r--r-- 1 root root   9707104 Mar 25 12:39 libmkl_sequential.a
-r-xr-xr-x 1 root root   8970003 Mar 25 12:39 libmkl_sequential.so
-r--r--r-- 1 root root      1048 Mar 25 09:05 libmkl_solver_ilp64.a
-r--r--r-- 1 root root      1048 Mar 25 09:05 libmkl_solver_ilp64_sequential.a
-r--r--r-- 1 root root      1048 Mar 25 09:05 libmkl_solver_lp64.a
-r--r--r-- 1 root root      1048 Mar 25 09:05 libmkl_solver_lp64_sequential.a
-r-xr-xr-x 1 root root   4266997 Mar 25 12:39 libmkl_vml_avx.so
-r-xr-xr-x 1 root root   2218467 Mar 25 12:39 libmkl_vml_def.so
-r-xr-xr-x 1 root root   3834953 Mar 25 12:39 libmkl_vml_mc2.so
-r-xr-xr-x 1 root root   3845965 Mar 25 12:39 libmkl_vml_mc3.so
-r-xr-xr-x 1 root root   3879774 Mar 25 12:39 libmkl_vml_mc.so
-r-xr-xr-x 1 root root   3325933 Mar 25 12:39 libmkl_vml_p4n.so
drwxr-xr-x 4 root root      4096 Jul 28 14:43 locale

--j
On Thu, Jul 29, 2010 at 2:46 PM, Stefan Theussl <stefan.theussl at wu.ac.at> wrote:
Dirk Eddelbuettel
#
On 29 July 2010 at 23:46, Stefan Theussl wrote:
| [...]
|  > P.S. If anyone has *easy* instructions for getting atlas working as a
|  > multithreaded library or how do drop in the Intel MKL (I couldn't
|  > figure out which *.so would be the correct one) please let me know and
|  > I'll post them as well.
| 
| I did get MKL working with R via the following trick:
| 
| 1) export 
| LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/intel/intel-mkl-9.0.018/lib/em64t
| 
| 2) then configure with: 
| "--with-blas=-L/opt/intel/intel-mkl-9.0.018/lib/em64t/ -lmkl -lguide 
| -lpthread"
| 
| Back then I used MKL version 9.0.018. I don't know if this still works 
| with more recent MKL or R.

Step 2) should not be necessary.  The BLAS is a binary interface; you can
"simply" drop the right ones in.  By having ld.so "find" the MKL ones before
the system ones, we made the MKL be used in the Revolution R release with
Ubuntu 9.10.  That still works for -- so no rebuilding of R needed here.
On 29 July 2010 at 14:56, Jonathan Greenberg wrote:
| Have you (or anyone) gotten Intel MKL working via the "plugin"
| approach that Dirk has setup for the debian packages, where you just
| point:


I have :)  

I have a proof-of-concept shell script that takes advantage of the Debian /
Ubuntu package management system and does the following for benchmarks:

    i)    removes all accelerates blas and runs with 'refblas'

    ii)   installs Atlas and runs with 'atlas'

    iii)  removes Atlas, install MKL (via aforementioned Revo package) and 
          runs with 'mkl'

    iv)   remove MKL and install goto (using .deb package built with the neat
          goto-helper package from the ISM in Japan) and run 'goto'
  
plus for kicks

    v)    install gputools and runs with 'gpu'

I showed first results in the 'Intro to HPC' tutorial at useR and want to do
some more work on this.  In fact, the gputools authors and I have a mostly
empty project on r-forge for this but not had time to do much...
Jonathan Greenberg
# 
I really like that all we have to do is "update-alternatives --install
/usr/lib/libblas.so.3gf" with your version -- gotoblas2 was a breeze
to get working this technique.  Mostly with MKL I'm just unclear on
what specific file I need to link against once its been installed.
Actually, along these lines, does LAPACK work independently from the
BLAS, e.g. will the "default" LAPACK (sudo apt-get install
liblapack-dev) use whichever BLAS happens to be linked up, or do we
need a different LAPACK (update-alternatives --config liblapack.so)
for each particularly BLAS we are using (e.g. use the GotoBLAS2 lapack
with GotoBLAS2, use Intel MKL's LAPACK, with Intel's BLAS, etc...)

And unfortunately no revolution-mkl for debian yet :(  The intel
install is easy tho, just run the install script and its all good.  I
did move it out of /opt tho and into /usr/lib to live with the rest of
the libraries on my system!

--j
On Thu, Jul 29, 2010 at 3:16 PM, Dirk Eddelbuettel <edd at debian.org> wrote:
Jonathan Greenberg
# 
Dirk:

Have you seen whether there is any significant performance boost in a
multithreaded system between GotoBLAS2 and Intel MKL?  I'm curious if
I should be spending time trying to get Intel MKL configured if I
already have a working GotoBLAS2...

--j

On Thu, Jul 29, 2010 at 3:29 PM, Jonathan Greenberg
<greenberg at ucdavis.edu> wrote:
Dirk Eddelbuettel
#
On 29 July 2010 at 15:29, Jonathan Greenberg wrote:
| And unfortunately no revolution-mkl for debian yet :(  The intel

The ones from Ubuntu seem to run ;-)

| install is easy tho, just run the install script and its all good.  I
| did move it out of /opt tho and into /usr/lib to live with the rest of
| the libraries on my system!

One word: "Don't". Moving files by hand into /usr (where dpkg and apt rule)
is simply a Very Bad Idea (TM).  Just say no.  Create a local deb, or simply
set up a /etc/ld.so.conf.d/localBlas.conf (look at other files there).
Dirk Eddelbuettel
#
On 29 July 2010 at 16:19, Jonathan Greenberg wrote:
| Have you seen whether there is any significant performance boost in a
| multithreaded system between GotoBLAS2 and Intel MKL?  I'm curious if
| I should be spending time trying to get Intel MKL configured if I
| already have a working GotoBLAS2...

On the (very, very limited) testing I have done they were at par.
Jonathan Greenberg
# 
I would completely agree about moving stuff except that there is no
.deb for Debian, so the Intel MKL I had was installed "by hand"
anyway, so no harm done, unless apt or dpkg decide to remove folders
they have no business touching.

Thanks for the info on goto vs. mkl, it looks like the dynamic linking
in MKL v. 10 is a bit different from v 9 -- in theory I should just
have to link up libmkl_core.so but I'm getting an error when using
this with R, I get an error as soon as I boot up R:

Error in dyn.load(file, DLLpath = DLLpath, ...) :
  unable to load shared library '/usr/lib64/R/library/stats/libs/stats.so':
  /usr/lib64/R/library/stats/libs/stats.so: undefined symbol: dswap_
During startup - Warning message:
package stats in options("defaultPackages") was not found

Here is my update-alternatives setting:

There are 4 choices for the alternative libblas.so.3gf (providing
/usr/lib/libblas.so.3gf).

  Selection    Path
Priority   Status
------------------------------------------------------------
  0            /usr/lib/atlas-corei7sse3/atlas/libblas.so.3gf
 60        auto mode
  1            /usr/lib/atlas-corei7sse3/atlas/libblas.so.3gf
 60        manual mode
  2            /usr/lib/gotoblas2/libgoto2_core2p-r1.13.so
 5         manual mode
* 3            /usr/lib/intel/mkl/10.2.5.035/lib/em64t/libmkl_core.so
 5         manual mode
  4            /usr/lib/libblas/libblas.so.3gf
 10        manual mode

We'll push ahead with gotoblas2, but if you have any ideas on
dynamically linking a non-deb version of MKL, please let me know!  Its
nice seeing all our processor lit up when using R!

--j
On Thu, Jul 29, 2010 at 4:25 PM, Dirk Eddelbuettel <edd at debian.org> wrote:
Ei-ji Nakama
# 
Hi

2010/7/30 Jonathan Greenberg <greenberg at ucdavis.edu>:
$ wget http://prs.ism.ac.jp/~nakama/debian/lenny-ism/gotoblas2-helper_0.1-12_all.deb
$ dpkg -i gotoblas2-helper_0.1-12_all.deb

register
$ iceweasel http://www.tacc.utexas.edu/?id=402

edit username and password
$ sudo vi /etc/gotoblas2-helper/gotoblas2-site.conf

download,dpkg-buildpackage and dpkg -i
$ sudo  /etc/init.d/gotoblas2-helper start

$ LANG=C ls -al  /usr/lib/gotoblas2/
total 33344
drwxr-xr-x   2 root root     4096 Feb  8 14:23 .
drwxr-xr-x 301 root root   143360 Jul 30 12:03 ..
-rw-r--r--   1 root root 11012466 Feb  8 14:22 libblas.a
lrwxrwxrwx   1 root root       14 Feb  8 14:23 libblas.so -> libblas.so.3gf
lrwxrwxrwx   1 root root       16 Feb  8 14:23 libblas.so.3gf ->
libblas.so.3gf.0
-rw-r--r--   1 root root  6057883 Feb  8 14:22 libblas.so.3gf.0
-rw-r--r--   1 root root  8825058 Feb  8 14:22 liblapack.a
lrwxrwxrwx   1 root root       16 Feb  8 14:23 liblapack.so -> liblapack.so.3gf
lrwxrwxrwx   1 root root       18 Feb  8 14:23 liblapack.so.3gf ->
liblapack.so.3gf.0
-rw-r--r--   1 root root  8033064 Feb  8 14:22 liblapack.so.3gf.0

$ ldd /usr/lib64/R/lib/libR.so | grep blas
	libblas.so.3gf => /usr/lib/gotoblas2/libblas.so.3gf (0x00007f2ebcea3000)