Skip to content

procedure to ship libomp.dylib run-time with package

12 messages · Timothy Bates, rsparapa at mcw.edu, Kasper Daniel Hansen +4 more

Original
Timothy Bates
# 
At https://mac.r-project.org/openmp/ it  says
"any package you compile against libomp.dylib will need that run-time so you have to ship it with your package or have users install it?

Are there  any  instructions on what to do/where to put libomp.dylib so it will be found?

I?ve compiled a package (OpenMx) in which OpenMP is working locally but other users get the error:

require(OpenMx)
Error: package or namespace load failed for ?OpenMx? in dyn.load(file, DLLpath = DLLpath, ...):
 unable to load shared object '/Users/***/Library/R/x86_64/4.4/library/OpenMx/libs/OpenMx.so':

dlopen(/Users/***/Library/R/x86_64/4.4/library/OpenMx/libs/OpenMx.so, 0x0006): Library not loaded: /Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/lib/libomp.dylib


The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannais a th? ann an Oilthigh Dh?n ?ideann, cl?raichte an Alba, ?ireamh cl?raidh SC005336.
rsparapa at mcw.edu
# 
Hi Tim:

/usr/local/lib it seems.  See the following page
https://mac.r-project.org/openmp/

--
Rodney Sparapani, Associate Professor of Biostatistics, He/Him/His
Vice President, Wisconsin Chapter of the American Statistical Association
Institute for Health and Equity, Division of Biostatistics
Medical College of Wisconsin, Milwaukee Campus

If this is outside of working hours, then please respond when convenient.

From: R-SIG-Mac <r-sig-mac-bounces at r-project.org> on behalf of Timothy Bates <tim.bates at ed.ac.uk>
Date: Monday, June 24, 2024 at 4:00?PM
To: R list <R-SIG-Mac at r-project.org>
Subject: [R-SIG-Mac] procedure to ship libomp.dylib run-time with package
ATTENTION: This email originated from a sender outside of MCW. Use caution when clicking on links or opening attachments.
________________________________

At https://urldefense.com/v3/__https://mac.r-project.org/openmp/__;!!H8mHWRdzp34!4HZ0uFxJQ121gF_TnX99OuMlgIXM600pq8wCbWruoLsdLTtaFgRB5_gMopCtxD0N9M4tUOO-7hBGADHx1sk$<https://urldefense.com/v3/__https:/mac.r-project.org/openmp/__;!!H8mHWRdzp34!4HZ0uFxJQ121gF_TnX99OuMlgIXM600pq8wCbWruoLsdLTtaFgRB5_gMopCtxD0N9M4tUOO-7hBGADHx1sk$>  it  says
"any package you compile against libomp.dylib will need that run-time so you have to ship it with your package or have users install it?

Are there  any  instructions on what to do/where to put libomp.dylib so it will be found?

I?ve compiled a package (OpenMx) in which OpenMP is working locally but other users get the error:

require(OpenMx)
Error: package or namespace load failed for ?OpenMx? in dyn.load(file, DLLpath = DLLpath, ...):
 unable to load shared object '/Users/***/Library/R/x86_64/4.4/library/OpenMx/libs/OpenMx.so':

dlopen(/Users/***/Library/R/x86_64/4.4/library/OpenMx/libs/OpenMx.so, 0x0006): Library not loaded: /Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/lib/libomp.dylib


The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannais a th? ann an Oilthigh Dh?n ?ideann, cl?raichte an Alba, ?ireamh cl?raidh SC005336.
_______________________________________________
R-SIG-Mac mailing list
R-SIG-Mac at r-project.org
https://urldefense.com/v3/__https://stat.ethz.ch/mailman/listinfo/r-sig-mac__;!!H8mHWRdzp34!4HZ0uFxJQ121gF_TnX99OuMlgIXM600pq8wCbWruoLsdLTtaFgRB5_gMopCtxD0N9M4tUOO-7hBGdICDldE$<https://urldefense.com/v3/__https:/stat.ethz.ch/mailman/listinfo/r-sig-mac__;!!H8mHWRdzp34!4HZ0uFxJQ121gF_TnX99OuMlgIXM600pq8wCbWruoLsdLTtaFgRB5_gMopCtxD0N9M4tUOO-7hBGdICDldE$>
Kasper Daniel Hansen
# 
If you read the next couple of sentences on that page, it says

The following are links to libomp OpenMP run-time built from official LLVM
release sources using Xcode compilers. They are signed and support macOS
10.13 (High Sierra) and higher. All tar-balls contain the system tree
usr/local/lib and usr/local/include so the recommended installation is to
type in Terminal:

    curl -O
https://mac.r-project.org/openmp/openmp-14.0.6-darwin20-Release.tar.gz
    sudo tar fvxz openmp-14.0.6-darwin20-Release.tar.gz -C /


I think that is pretty clear? Note the -C / as part of the last line, that
tells tar where to put it on the system.

On Tue, Jun 25, 2024 at 8:20?AM Sparapani, Rodney via R-SIG-Mac <
r-sig-mac at r-project.org> wrote:

            

      
      
      
    

        

      
      
    

        

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Timothy Bates
    

    

      
      #
    

  


  

      
Thanks Kasper,
I read those lines and had installed the openmp software as instructed. It all works fine for me. But users don?t have my usr/ folder, not can a package write to their usr/ folder.

so what is confusing is how the user will get this lib. Where should make place libraries like libomp within the package so that if a user merely install.packages() it, the library will be found and loaded?
Best, tim


Sent from Outlook for iOS<https://aka.ms/o0ukef>

        

      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Balamuta, James Joseph
    

    

      
      #
    

  


  

      
Tim,

Unfortunately, there isn't a nice way of including OpenMP for macOS within a package.  One potential avenue would be to use a `configure.ac` file to check for whether OpenMP headers are present on the macOS computer and, then, allow the correct OpenMP compilation flags to be set. However, this requires the user to compile from source instead of using a CRAN binary.

Unfortunately, there isn't going to be an easier way as OpenMP on macOS with R is experimental by nature.

Best regards,

James

        

      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Bram Boeckx
    

    

      
      #
    

  


  

      
Dear,

Thanks for your message,
I'm currently on paternity leave until 23th of July and cannot respond to your email. Please contact my colleagues if you have questions or need assistance.

Best Wishes,

Bram

        
On 25 Jun 2024, at 14:20, Sparapani, Rodney via R-SIG-Mac <R-SIG-Mac at r-project.org> wrote:

        
Hi Tim:

/usr/local/lib it seems.  See the following page
https://mac.r-project.org/openmp/

--
Rodney Sparapani, Associate Professor of Biostatistics, He/Him/His
Vice President, Wisconsin Chapter of the American Statistical Association
Institute for Health and Equity, Division of Biostatistics
Medical College of Wisconsin, Milwaukee Campus

If this is outside of working hours, then please respond when convenient.

From: R-SIG-Mac <r-sig-mac-bounces at r-project.org> on behalf of Timothy Bates <tim.bates at ed.ac.uk>
Date: Monday, June 24, 2024 at 4:00?PM
To: R list <R-SIG-Mac at r-project.org>
Subject: [R-SIG-Mac] procedure to ship libomp.dylib run-time with package
ATTENTION: This email originated from a sender outside of MCW. Use caution when clicking on links or opening attachments.
________________________________

At https://urldefense.com/v3/__https://mac.r-project.org/openmp/__;!!H8mHWRdzp34!4HZ0uFxJQ121gF_TnX99OuMlgIXM600pq8wCbWruoLsdLTtaFgRB5_gMopCtxD0N9M4tUOO-7hBGADHx1sk$<https://urldefense.com/v3/__https:/mac.r-project.org/openmp/__;!!H8mHWRdzp34!4HZ0uFxJQ121gF_TnX99OuMlgIXM600pq8wCbWruoLsdLTtaFgRB5_gMopCtxD0N9M4tUOO-7hBGADHx1sk$>  it  says
"any package you compile against libomp.dylib will need that run-time so you have to ship it with your package or have users install it?

Are there  any  instructions on what to do/where to put libomp.dylib so it will be found?

I?ve compiled a package (OpenMx) in which OpenMP is working locally but other users get the error:

require(OpenMx)
Error: package or namespace load failed for ?OpenMx? in dyn.load(file, DLLpath = DLLpath, ...):
unable to load shared object '/Users/***/Library/R/x86_64/4.4/library/OpenMx/libs/OpenMx.so':

dlopen(/Users/***/Library/R/x86_64/4.4/library/OpenMx/libs/OpenMx.so, 0x0006): Library not loaded: /Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/lib/libomp.dylib


The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannais a th? ann an Oilthigh Dh?n ?ideann, cl?raichte an Alba, ?ireamh cl?raidh SC005336.
_______________________________________________
R-SIG-Mac mailing list
R-SIG-Mac at r-project.org
https://urldefense.com/v3/__https://stat.ethz.ch/mailman/listinfo/r-sig-mac__;!!H8mHWRdzp34!4HZ0uFxJQ121gF_TnX99OuMlgIXM600pq8wCbWruoLsdLTtaFgRB5_gMopCtxD0N9M4tUOO-7hBGdICDldE$<https://urldefense.com/v3/__https:/stat.ethz.ch/mailman/listinfo/r-sig-mac__;!!H8mHWRdzp34!4HZ0uFxJQ121gF_TnX99OuMlgIXM600pq8wCbWruoLsdLTtaFgRB5_gMopCtxD0N9M4tUOO-7hBGdICDldE$>


_______________________________________________
R-SIG-Mac mailing list
R-SIG-Mac at r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-mac

  
  


  


      

    

    

      
      

        


  

        

      Brian Ripley
    

    

      
      #
    

  


  

      
On 26/06/2024 06:41, Balamuta, James wrote:

            

      
      
      
    

        
Package installation also requires the libomp.dylib, and to know where 
to find it both when linking and loading.

            

      
      
      
    

        
Rather, macOS makes it deliberately hard, but those instructions are not 
current.

- The standard way for the macOS linker to record paths to a dependent 
dylib is as an absolute path.  E.g. (arm64)

% otool -L library/stats/libs/stats.so
library/stats/libs/stats.so:
	stats.so (compatibility version 0.0.0, current version 0.0.0)
	/Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/lib/libRlapack.dylib (compatibility version 4.4.0, current version 4.4.1)
...

That is a location that is not writeable to an end user.  However, it 
appears that R 4.4.1 for arm64 ships with a bi-arch libomp.dylib there.

auk2% file `R RHOME`/lib/libomp.dylib
/Library/Frameworks/R.framework/Resources/lib/libomp.dylib: Mach-O 
universal binary with 2 architectures: [x86_64:Mach-O 64-bit dynamically 
linked shared library x86_64] [arm64:Mach-O 64-bit dynamically linked 
shared library arm64]
/Library/Frameworks/R.framework/Resources/lib/libomp.dylib (for 
architecture x86_64):	Mach-O 64-bit dynamically linked shared library x86_64
/Library/Frameworks/R.framework/Resources/lib/libomp.dylib (for 
architecture arm64):	Mach-O 64-bit dynamically linked shared library arm64

So it is likely that libomp.dylib is already available somewhere that 
will be searched at both link and load times.  (omp.h needs to be put in 
a suitable place, e.g. /usr/local/include for Intel, but that needs 
admin privileges. That is only needed at compilation, so could be put 
anywhere and -I specified.)

- Another issue: That page provides multiple copies on libomp.dylib: how 
could package installation know which is appropriate?  I do not know 
which one was shipped with R, nor how find out.  But it suggests 
shipping one with a package would be unsafe.

It seems the OP asked the wrong question: apparently he distributed a 
package built with R 4.3.x, and someone tried to use it on 4.4.x.  R 
binary packages are tied to an R x.y series (as the manual would have 
told you).

I was able to build OpenMx with OpenMP on arch64 by using

PKG_CPPFLAGS = -Xclang -fopenmp -I/path/to/omp.h

PKG_LIBS= -lomp \
...

(Caveat: for OpenMx, <omp.h> is used so the -I adds to the system 
include path and there is danger of conflicts. Best to use a separate 
directory for the OpenMP headers.)

And please comply with the posting guide, not send HTML and not mangle URLs.

            

      
      
      
    

        

      
      
    

        
There is a version mismatch here ....

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Bram Boeckx
    

    

      
      #
    

  


  

      
Dear,

Thanks for your message,
I'm currently on paternity leave until 23th of July and cannot respond to your email. Please contact my colleagues if you have questions or need assistance.

Best Wishes,

Bram

        
On 26 Jun 2024, at 11:57, Prof Brian Ripley via R-SIG-Mac <R-SIG-Mac at r-project.org> wrote:

        

        
On 26/06/2024 06:41, Balamuta, James wrote:

        
Tim,
Unfortunately, there isn't a nice way of including OpenMP for macOS within a package.  One potential avenue would be to use a `configure.ac` file to check for whether OpenMP headers are present on the macOS computer and, then, allow the correct OpenMP compilation flags to be set. However, this requires the user to compile from source instead of using a CRAN binary.

Package installation also requires the libomp.dylib, and to know where to find it both when linking and loading.

Unfortunately, there isn't going to be an easier way as OpenMP on macOS with R is experimental by nature.

Rather, macOS makes it deliberately hard, but those instructions are not current.

- The standard way for the macOS linker to record paths to a dependent dylib is as an absolute path.  E.g. (arm64)

% otool -L library/stats/libs/stats.so
library/stats/libs/stats.so:
        stats.so (compatibility version 0.0.0, current version 0.0.0)
        /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/lib/libRlapack.dylib (compatibility version 4.4.0, current version 4.4.1)
...

That is a location that is not writeable to an end user.  However, it appears that R 4.4.1 for arm64 ships with a bi-arch libomp.dylib there.

auk2% file `R RHOME`/lib/libomp.dylib
/Library/Frameworks/R.framework/Resources/lib/libomp.dylib: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit dynamically linked shared library x86_64] [arm64:Mach-O 64-bit dynamically linked shared library arm64]
/Library/Frameworks/R.framework/Resources/lib/libomp.dylib (for architecture x86_64):   Mach-O 64-bit dynamically linked shared library x86_64
/Library/Frameworks/R.framework/Resources/lib/libomp.dylib (for architecture arm64):    Mach-O 64-bit dynamically linked shared library arm64

So it is likely that libomp.dylib is already available somewhere that will be searched at both link and load times.  (omp.h needs to be put in a suitable place, e.g. /usr/local/include for Intel, but that needs admin privileges. That is only needed at compilation, so could be put anywhere and -I specified.)

- Another issue: That page provides multiple copies on libomp.dylib: how could package installation know which is appropriate?  I do not know which one was shipped with R, nor how find out.  But it suggests shipping one with a package would be unsafe.

It seems the OP asked the wrong question: apparently he distributed a package built with R 4.3.x, and someone tried to use it on 4.4.x.  R binary packages are tied to an R x.y series (as the manual would have told you).

I was able to build OpenMx with OpenMP on arch64 by using

PKG_CPPFLAGS = -Xclang -fopenmp -I/path/to/omp.h

PKG_LIBS= -lomp \
...

(Caveat: for OpenMx, <omp.h> is used so the -I adds to the system include path and there is danger of conflicts. Best to use a separate directory for the OpenMP headers.)

And please comply with the posting guide, not send HTML and not mangle URLs.

Best regards,
James

        

      
      
    

  
  


  


      

    

    
1 day later

    

      
      

        


  

        

      Simon Urbanek
    

    

      
      #
    

  


  

      
Tim,

There is no need to worry about libomp.dylib on macOS with CRAN packages as libomp is already shipped with R. When the package its built on CRAN, it will be pointed correctly to the version inside R.

However, in your output it seems that the problem may be that the user is using the wrong R version - the package is installed for R 4.4, but it seems to be looking for libomp in R 4.3. They are compatible, but the paths look wrong.  It is not clear to me what paths you used when building the package binary. Note that the package can only be installed for the same version that it has been compiled for. Also if you compile against libomp you have to either link against the version in R or use install_name_tool to change the linking path to make sure it points to the R version. Either way, check with tool -L to see what you are linking to.

Cheers,
Simon

            

      
      
      
    

        

      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Simon Urbanek
    

    

      
      #
    

  


  

      
Most of the claims below are false - libomp is shipped with CRAN R for quite some time specifically so that packages which benefit from it significantly can enable OpenMP reasonably easily - they only need to add the flags as noted on the cited page and CRAN takes care of the rest. If there are any problems, please feel free to contact me.

Cheers,
Simon

            

      
      
      
    

        

      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Balamuta, James Joseph
    

    

      
      #
    

  


  

      
Simon,

I think there is a communication issue regarding OpenMP for the macOS R community. That said, I?m both thrilled and delighted that I?m wrong in this case! To my knowledge, there was never a public facing note regarding `libomp`  shipping with CRAN?s official R binary for mac, [1]. In fact, the last tracking I have with respect to OpenMP and R on macOS was its abrupt and painful removal that led to the instructions for experimental inclusion of OpenMP on mac within the developer website [2] (gently acknowledge by Prof. Ripley as being not current). This has led to a lot of lobotomization in compiled code regarding macOS that can now hopefully be adjusted for the multi-core world, [3].

With this in mind, I suppose this means that we would need to revisit the `configure.ac` [3]. Potentially, as Prof. Ripley mentioned, we?re looking at including on path/detecting:

`R RHOME`/lib/libomp.dylib/Library/Frameworks/R.framework/Resources/lib/libomp.dylib

Thus, the package binary downloaded from CRAN would correctly set and detect cores per:

# install.packages("RcppArmadillo")

# Determine the number of active threads
RcppArmadillo::armadillo_get_number_of_omp_threads()
[1] 1

# Set number of threads
RcppArmadillo::armadillo_set_number_of_omp_threads(3)

# Retrieve OpenMP threads after setting
RcppArmadillo::armadillo_get_number_of_omp_threads()
[1] 1
# This should be 3.

I?m happy to talk more on the list or off the list.

Best,

James

[1]: https://cran.r-project.org/doc/manuals/r-release/NEWS.html
[2]: https://mac.r-project.org/openmp/
[3]: https://github.com/RcppCore/RcppArmadillo/blob/37461ba36472305c699263afc229919d37daa5e3/configure.ac#L109


From: Simon Urbanek <simon.urbanek at R-project.org>
Date: Thursday, June 27, 2024 at 10:25?AM
To: Balamuta, James <balamut2 at illinois.edu>
Cc: Timothy Bates <tim.bates at ed.ac.uk>, Kasper Daniel Hansen <kasperdanielhansen at gmail.com>, Sparapani, Rodney <rsparapa at mcw.edu>, R list <R-SIG-Mac at r-project.org>
Subject: Re: [R-SIG-Mac] procedure to ship libomp.dylib run-time with package
Most of the claims below are false - libomp is shipped with CRAN R for quite some time specifically so that packages which benefit from it significantly can enable OpenMP reasonably easily - they only need to add the flags as noted on the cited page and CRAN takes care of the rest. If there are any problems, please feel free to contact me.

Cheers,
Simon

            

      
      
      
    

        

      
      
    

        

  
  


  


      

    

    

      
      

        


  

        

      Simon Urbanek
    

    

      
      #
    

  


  

          

      
      
      
    

        
That's entirely unnecessary - you should not be requiring specific file paths - that's not how linking tests work. All the instructions are on the OpenMP page for years - it is as simple as -lomp (which is standard almost everywhere). The only non-standard requirement compared to other platforms is that instead of -fopemp in C flags the package should also check for -Xclang -fopenmp.

If a package explicitly disables OpenMP for Apple compilers, it is its deliberate choice - it was never necessary. The whole point of configure scripts is to detect functionality by testing it and use it if available.

Cheers,
Simon