[Bioc-devel] xps build problem on veracruz2

Disabling SIP should not be done anywhere. Every page I've read on this topic strongly discourages doing this.


----- Original Message -----

From: "Herv? Pag?s" <hpages at fredhutch.org>
To: "cstrato" <cstrato at aon.at>, "Dan Tenenbaum" <dtenenba at fredhutch.org>
Cc: "bioc-devel" <bioc-devel at r-project.org>
Sent: Thursday, April 20, 2017 3:17:23 PM
Subject: Re: [Bioc-devel] xps build problem on veracruz2

On 04/20/2017 03:01 PM, cstrato wrote:

Dear Herve,

Doing 'csrutil disable' does indeed solve the problem, since:

1, in this way I was able to build xps on Mac OS Sierra

2, in this way I could already help one user of xps, see:
https://urldefense.proofpoint.com/v2/url?u=https-3A__support.bioconductor.org_p_90056_-2390247&d=DwIDaQ&c=eRAMFD45gAfqt84VtBcfhQ&r=BK7q3XeAvimeWdGbWY_wJYbW0WYiZvSXAJJKaaPhzWA&m=q0hKDI_veZEYmjrYbsK1Xqu--fGdfl_JmfSKLygl_dg&s=GfzgU1Ibm_scFWO58Mv_ZfxKtn-FSJgkxMW1ZBYK1Vs&e=

This means, that users of xps seem to be willing to disable csrutil.

I'm not a statistician but I wouldn't draw conclusions from a sample
of size 1. Mac users who cannot install xps on their machine will
use something else, or, if they desperately need xps, they will
grab a Linux box.

Sorry but disabling SIP on our Mac builders is not an option.

However, I just realized that there may be an even greater problem,
namely, which version of ROOT should a user install?

People can download ROOT binaries built with XCode 7.x from:
https://urldefense.proofpoint.com/v2/url?u=https-3A__root.cern.ch_content_release-2D53436&d=DwIDaQ&c=eRAMFD45gAfqt84VtBcfhQ&r=BK7q3XeAvimeWdGbWY_wJYbW0WYiZvSXAJJKaaPhzWA&m=q0hKDI_veZEYmjrYbsK1Xqu--fGdfl_JmfSKLygl_dg&s=LByiYPyOfsHnRc9oOqXOs9xMcfltktXgkTQ3sh5hKFc&e=
for OS X 10.10 and 10.11.

Thus for El Capitan they can download the following binaries:
- root_v5.34.36.macosx64-10.11-clang70.dmg
- root_v5.34.36.macosx64-10.11-clang70.tar.gz

Did you install one of those binaries or did you compile ROOT from source?

As I said earlier, I compiled ROOT 5 from source on veracruz2.

With the help of my README file people could compile ROOT from source
for XCode 8.x.


However, you have mentioned that the CRAN people are using clang 4.0.0
for producing the Mac binaries of R and CRAN packages and thus you are
using the same on veracruz2.

Yes.

Did you compile ROOT with XCode 7 or 8 or did you use clang 4.0.0, which
is not officially supported by Apple?

With clang 4.0.0.

The question is whether xps built with this version of ROOT will work
with the ROOT binaries which people can download from ROOT?

I guess someone will need to figure this out.

Note that if people need to compile their own ROOT anyway in order to
be able to use the xps binary we distribute, then they should also be
able to install xps from source. So that defeats the purpose of
providing a binary in the first place.

Cheers,
H.

Best regards,
Christian



On 04/20/17 20:00, Herv? Pag?s wrote:

On 04/20/2017 10:59 AM, Herv? Pag?s wrote:

Hi Christian,

Disabling 'csrutil disable' might help xps on veracruz2 but that

  ^^^^^^^^^^^^^^^^^^^^^^^^^^
oops, no double negative intended here. I meant, doing 'csrutil disable'
might help... etc

H.

won't help your end users.

I'm no expert in developing a package on Mac but other people on
this list are. Also R-SIG-Mac might be a good place to seek help
for this.

Cheers,
H.


On 04/20/2017 10:53 AM, cstrato wrote:

Dear Herve,

Thank you for your efforts to try to install xps. I am glad to hear
that
you could build ROOT 5 from source.

It's a pity that Apple does no longer allow the use of
DYLD_LIBRARY_PATH. This seems to break the code of a lot of people
(when
googling around).

I will try to change the build process and will see if I succeed.
However, at the moment I have a couple of questions:


1, Is there any reason that you do not want to 'csrutil disable'?


2, It is easy to delete the test for DYLD_LIBRARY_PATH in my
'configure.in' file, however, I have also to change the 'Makefile'.

Since you say that the problem can be solved by adding the -rpath flag
when linking, I assume that I have to change  the following line in my
'Makefile':
        $(LD) -bundle $(LDFLAGS) $^ $(GLIBS) $(MYLIBS) \
           $(OutPutOpt) $(subst .$(DllSuf),.so,$@)

Since I am not familiar with -rpath can you give me a hint how to
change
it?


3, There is a Wikipedia explanation for '-rpath', see:
https://urldefense.proofpoint.com/v2/url?u=https-3A__en.wikipedia.org_wiki_Rpath&d=DwIDaQ&c=eRAMFD45gAfqt84VtBcfhQ&r=BK7q3XeAvimeWdGbWY_wJYbW0WYiZvSXAJJKaaPhzWA&m=kxgD4oa4CZeT4T7uLq2m3iVTvgvFB_RKN1Rf1KcxPk0&s=ge_d4eBoKDAggNWVQUzMVPAJy250VlZuPTXcPyr20HM&e=




Interestingly, there is the following line:

'Instead of specifying the -rpath to the linker, the environment
variable LD_RUN_PATH can be set to the same effect.'

Do you think that using LD_RUN_PATH would solve the problem?
If yes, then how?


4, Googling for the DYLD_LIBRARY_PATH problem I have also found the
following discussion:
https://urldefense.proofpoint.com/v2/url?u=https-3A__forums.macrumors.com_threads_is-2Dit-2Dok-2Dto-2Duse-2Ddyld-5Flibrary-5Fpath-2Don-2Dmac-2Dos-2Dx-2Dand-2Dwhat-25C2-2592s-2Dthe-2Ddynamic-2Dlibrary-2Dsearch.956258_&d=DwIDaQ&c=eRAMFD45gAfqt84VtBcfhQ&r=BK7q3XeAvimeWdGbWY_wJYbW0WYiZvSXAJJKaaPhzWA&m=kxgD4oa4CZeT4T7uLq2m3iVTvgvFB_RKN1Rf1KcxPk0&s=yhLXARIG_RS1LNegc7tbjO1bayLOp7zy5-rzbtjCHIk&e=




There, one answer mentioned:
'Also, rpath is a good idea. Also see install_name_tool, which can
change the load paths of libraries.'

Doing a search for 'install_name_tool' I found:
https://urldefense.proofpoint.com/v2/url?u=http-3A__stackoverflow.com_questions_2985315_using-2Dinstall-2Dname-2Dtool-2Dwhats-2Dgoing-2Dwrong&d=DwIDaQ&c=eRAMFD45gAfqt84VtBcfhQ&r=BK7q3XeAvimeWdGbWY_wJYbW0WYiZvSXAJJKaaPhzWA&m=kxgD4oa4CZeT4T7uLq2m3iVTvgvFB_RKN1Rf1KcxPk0&s=r5LiHlxkGAeJAcciwteD2XD6ffNtWLiknvcIj9EJL8E&e=




There, one answer mentioned (see also $man install_name_tool):
'Having experimented more: install_name_tool -id newname file will do
the trick.'

See also:
https://urldefense.proofpoint.com/v2/url?u=http-3A__qin.laya.com_tech-5Fcoding-5Fhelp_dylib-5Flinking.html&d=DwIDaQ&c=eRAMFD45gAfqt84VtBcfhQ&r=BK7q3XeAvimeWdGbWY_wJYbW0WYiZvSXAJJKaaPhzWA&m=kxgD4oa4CZeT4T7uLq2m3iVTvgvFB_RKN1Rf1KcxPk0&s=FL17xv1uuzR6UF59mgJstMQ-1seE1UeKDQVVXQgXnUo&e=





Do you think that either using the environment variable LD_RUN_PATH or
using 'install_name_tool' could solve the problem without having to use
'-rpath'?


Thank you in advance.
Best regards,
Christian



On 04/19/17 22:51, Herv? Pag?s wrote:

Hi Christian,

So I installed ROOT 5 from source on veracruz2. It's in
/usr/local/root.

However, Apple's SIP (System Integrity Protection, new and
enabled by default on El Capitan) is getting in the way when
trying to install xps. That's because xps configure and build
process relies on DYLD_LIBRARY_PATH. Problem is that this
environment variable (and any other variables that control
dynamic loading) is not inherited by child processes when SIP
is on:

veracruz2:~ biocbuild$ if test "${DYLD_LIBRARY_PATH}"; then echo
'yep!';
else echo 'nope!'; fi
yep!

veracruz2:~ biocbuild$ sh
sh-3.2$ if test "${DYLD_LIBRARY_PATH}"; then echo 'yep!'; else echo
'nope!'; fi
nope!

That breaks xps configure script:

veracruz2:~ biocbuild$ export LD_LIBRARY_PATH=$DYLD_LIBRARY_PATH

veracruz2:~ biocbuild$ echo $LD_LIBRARY_PATH
/usr/local/mysql/lib:/usr/local/root/lib/root:/ImageMagick-7.0.5/lib:/usr/local/ensembl-vep/htslib






veracruz2:~ biocbuild$ R CMD INSTALL xps
* installing to library
?/Library/Frameworks/R.framework/Versions/3.4/Resources/library?
* installing *source* package ?xps? ...
checking for gcc... clang
checking for C compiler default output file name... a.out
checking whether the C compiler works... yes
checking whether we are cross compiling... no
checking for suffix of executables...
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether clang accepts -g... yes
checking for clang option to accept ANSI C... none needed
checking how to run the C preprocessor... clang -E
checking for gcc... (cached) clang
checking whether we are using the GNU C compiler... (cached) yes
checking whether clang accepts -g... (cached) yes
checking for clang option to accept ANSI C... (cached) none needed
found ROOT version 5.34/36 in directory /usr/local/root

xps configuration error:

   You must set the shell variable LD_LIBRARY_PATH to the
   directory where ROOT resides and re-run R CMD INSTALL
   e.g., (using Bourne shell syntax):

      export "LD_LIBRARY_PATH=$ROOTSYS/lib:$LD_LIBRARY_PATH"
      R CMD INSTALL xps

   Please consult the README file for more information

ERROR: configuration failed for package ?xps?
* removing
?/Library/Frameworks/R.framework/Versions/3.4/Resources/library/xps?
* restoring previous
?/Library/Frameworks/R.framework/Versions/3.4/Resources/library/xps?

That also breaks the dynlib mechanism because, after I managed to
produce xps.so, it turns out that this shared object is linked to
the ROOT libraries via the @rpath mechanism:

veracruz2:src biocbuild$ otool -L xps.so
xps.so:
    xps.so (compatibility version 0.0.0, current version 0.0.0)
    @rpath/libGui.so (compatibility version 0.0.0, current version
0.0.0)
    @rpath/libCore.so (compatibility version 0.0.0, current version
0.0.0)
    @rpath/libCint.so (compatibility version 0.0.0, current version
0.0.0)
    @rpath/libRIO.so (compatibility version 0.0.0, current version
0.0.0)
    @rpath/libNet.so (compatibility version 0.0.0, current version
0.0.0)
    @rpath/libHist.so (compatibility version 0.0.0, current version
0.0.0)
    @rpath/libGraf.so (compatibility version 0.0.0, current version
0.0.0)
    @rpath/libGraf3d.so (compatibility version 0.0.0, current version
0.0.0)
    @rpath/libGpad.so (compatibility version 0.0.0, current version
0.0.0)
    @rpath/libTree.so (compatibility version 0.0.0, current version
0.0.0)
    @rpath/libRint.so (compatibility version 0.0.0, current version
0.0.0)
    @rpath/libPostscript.so (compatibility version 0.0.0, current
version 0.0.0)
    @rpath/libMatrix.so (compatibility version 0.0.0, current version
0.0.0)
    @rpath/libPhysics.so (compatibility version 0.0.0, current version
0.0.0)
    @rpath/libMathCore.so (compatibility version 0.0.0, current
version
0.0.0)
    @rpath/libThread.so (compatibility version 0.0.0, current version
0.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version 1226.10.1)
    @rpath/libGed.so (compatibility version 0.0.0, current version
0.0.0)
    @rpath/libTreePlayer.so (compatibility version 0.0.0, current
version 0.0.0)
    @rpath/libTreeViewer.so (compatibility version 0.0.0, current
version 0.0.0)
    /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current
version 120.1.0)

so it can't be loaded in R:

dyn.load("xps.so")

Error in dyn.load("xps.so") :
  unable to load shared object
'/Users/biocbuild/bbs-3.5-bioc/xps/src/xps.so':
  dlopen(/Users/biocbuild/bbs-3.5-bioc/xps/src/xps.so, 6): Library not
loaded: @rpath/libGui.so
  Referenced from: /Users/biocbuild/bbs-3.5-bioc/xps/src/xps.so
  Reason: image not found

because R has no access to DYLD_LIBRARY_PATH:

Sys.getenv("DYLD_LIBRARY_PATH")

[1] ""

This can be addressed by adding the following flag when linking:

  -rpath $(shell $(ROOTCONFIG) --prefix)/lib

Do you think you can revisit xps configure and build process? Make
sure
you test it on a machine where SIP is enabled.

Thanks,
H.

 snip >>>

On 03/23/17 17:47, Herv? Pag?s wrote:

Hi Christian,

The CRAN folks are currently experimenting with clang 4.0.0
for
producing the Mac binaries of R and CRAN packages so we are
using
the same on veracruz2. This is a version of clang that is
ahead of
what's in XCode 8.x or XCode 7.x. So I guess that means
we'll
have
to compile ROOT from source on veracruz2.

BTW any reason not to make xps work with ROOT 6?

Cheers,
H.

On 03/23/2017 07:28 AM, cstrato wrote:

Dear Valerie,

I have seen that you have set up a new Mac server,
veracruz2,
running El
Capitan.

Although the development version of xps does even run on
Mac OS
Sierra,
one issue still remains the same:

You need to install the latest ROOT version 5, since xps
does
not
run
with ROOT 6!

So you need to install on veracruz2 the same root version
that
you
have
installed on toluca2 running Maverics, i.e.
root_v5.34.36.macosx64-10.11-clang70.dmg

However, if you have installed on El Capitan XCode 8.x
instead of
XCode
7.x, then you need to compile ROOT from source, i.e.:
https://urldefense.proofpoint.com/v2/url?u=https-3A__root.cern.ch_download_root-5Fv5.34.36.source.tar.gz&d=DwICAg&c=eRAMFD45gAfqt84VtBcfhQ&r=BK7q3XeAvimeWdGbWY_wJYbW0WYiZvSXAJJKaaPhzWA&m=q9mk6yIytaNZlSdiLX_dFwchX8Tb7ra6x3WBBNIcs2o&s=Lz7YkqZ3XwjRsYIXVTbSvbDvTM-jTyoWvoVSa1PdBDw&e=







The README file of xps does explain how to compile ROOT
for
Sierra. This
should also be valid for El Capitan running XCode 8.x.

Thank you in advance.
Best regards,
Christian
_._._._._._._._._._._._._._._._._._
C.h.r.i.s.t.i.a.n   S.t.r.a.t.o.w.a
V.i.e.n.n.a           A.u.s.t.r.i.a
e.m.a.i.l:        cstrato at aon.at
_._._._._._._._._._._._._._._._._._

--
Herv? Pag?s

Program in Computational Biology
Division of Public Health Sciences
Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N, M1-B514
P.O. Box 19024
Seattle, WA 98109-1024

E-mail: hpages at fredhutch.org
Phone:  (206) 667-5791
Fax:    (206) 667-1319

--
Herv? Pag?s

Program in Computational Biology
Division of Public Health Sciences
Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N, M1-B514
P.O. Box 19024
Seattle, WA 98109-1024

E-mail: hpages at fredhutch.org
Phone:  (206) 667-5791
Fax:    (206) 667-1319