2 versions of same library loaded
Comments/questions interspersed below.
On Wed, 2014-03-12 at 22:50 -0400, Simon Urbanek wrote:
Ross, On Mar 12, 2014, at 5:34 PM, Ross Boylan <ross at biostat.ucsf.edu> wrote:
Can anyone help me understand how I got 2 versions of the same library loaded, how to prevent it, and what the consequences are? Running under Debian GNU/Linux squeeze. lsof and /proc/xxx/map both show 2 copies of several libraries loaded: /home/ross/install/lib/libmpi.so.1.3.0 /home/ross/install/lib/libopen-pal.so.6.1.0 /home/ross/install/lib/libopen-rte.so.7.0.0 /home/ross/Rlib-3.0.1/Rmpi/libs/Rmpi.so /usr/lib/openmpi/lib/libmpi.so.0.0.2 /usr/lib/openmpi/lib/libopen-pal.so.0.0.0 /usr/lib/openmpi/lib/libopen-rte.so.0.0.0 /usr/lib/R/lib/libR.so The system has the old version of MPI installed under /usr/lib. I built a personal, newer copy in my directory, and then rebuilt Rmpi (an R package) against it. ldd on the personal Rmpi.so and libmpi.so shows all references to mpi libraries on personal paths. R was installed from a debian package, and presumably compiled without having MPI around. Before running I set LD_LIBRARY_PATH to ~/install/lib, and then stuffed the same path at the start of LD_LIBRARY_PATH using Sys.setenv in my profile because R seems to prepend some libraries to that path when it starts (I'm curious about that too). I also prepended ~/install/bin to my path, though I'm not sure that's relevant. Does R use ld.so or some other mechanism for loading libraries?
R uses dlopen to load package libraries - it is essentially identical to using ld.so for dependencies.
Can I assume the highest version number of a library will be preferred?
No.
Bummer. The fact that Rmpi is not crashing suggests to me it's using the right version of the mpi libraries (it does produce lots of errors if I run it without setting LD_LIBRARY_PATH so only the system mpi libs are in play), but it would be nice to be certain. Or the 2 versions could be combined in a big mess.
http://cran.r-project.org/doc/manuals/r-devel/R-exts.html#index-Dynamic-loading says "If a shared object/DLL is loaded more than once the most recent version is used." I'm not sure if "most recent" means the one loaded most recently by the program (I don't know which that is) or "highest version number."
The former - whichever you load last wins. Note, however, that this refers to explicitly loaded objects since they are loaded into a flat namespace so a load will overwrite all symbols that get loaded.
It might be good to clarify that in the manual. If I understand the term, the mpi libraries are loaded implicitly; that is, Rmpi.so is loaded explicitly, and then it pulls in dependencies. What are the rules in that case?
Why is /usr/lib/openmpi being looked at in the first place?
You'll have to consult your system. The search path (assuming rpath is not involved) is governed by LD_LIBRARY_PATH and /etc/ld.so.conf*. Note that LD_LIBRARY_PATH is consulted at the time of the resolution (when the library is looked up), so you may be changing it too late. Also note that you have to expand ~ in the path (it's not a valid path, it's a shell expansion feature).
I just used the ~ as a shortcut; the shell expanded it and the full path ended up in the variable. I assume the loader checks LD_LIBRARY_PATH first; once it finds the mpi libraries there I don't know why it keeps looking. I'm not sure I follow the part about too late, but is it this?: all the R's launched under MPI have the MPI library loaded automatically. If that happens before my profile is read, reseting LD_LIBRARY_PATH will be irrelevant. I don't know whether the profile or Rmpi load happens first. The reseting is just a reordering, and since the other elements in LD_LIBRARY_PATH don't have any mpi libraries I don't think the order matters.
R's massaging of the LD_LIBRARY_PATH is typically done in $R_HOME/etc/ldpaths so you may want to check it and/or adjust it as needed. Normally (in stock R), it only prepends its own libraries and Java so it should not be causing any issues, but you may want to check in case Debian scripts add anything else.
The extra paths are limited as you describe, and so are probably no threat for loading the wrong MPI library (/usr/lib64/R/lib:/usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server).
How can I stop the madness? Some folks on the openmpi list have indicated I need to rebuild R, telling it where my MPI is, but that seems an awfully big hammer for the problem.
I would check LD_LIBRARY_PATH and also check at which point are those old libraries loaded to find where they are referenced.
How do I tell the point at which the old libraries are loaded? I assume it happens implicitly when Rmpi is loaded, but I don't know which of the 2 versions of the libraries is loaded first, and I don't know how to tell. Thanks for your help. Ross Boylan