Skip to content

openmp and xcode

6 messages · Simon Urbanek, biii+rsigm@c m@iii@g oii 8p@wexpress@com, Taras Zakharko

Original
biii+rsigm@c m@iii@g oii 8p@wexpress@com
# 
If the CRAN R installation for macos includes |libomp.dylib|, is there a 
reason it does not include the paired |./include/omp*.h| files as well? 
It?s just three, they?re small, and it seems a reasonable step to making 
building packages that use OpenMP more direct.

Background: I?ve been having some difficulty compiling |data.table| 
(dev) on my MBP (M4, 15.4.1, R-4.4.3), due to an error (I do not assume 
it?s this package, I?ve had trouble with other packages too).

|symbol not found in flat namespace '___kmpc_dispatch_deinit' |

When troubleshooting this, I?ve verified that I have Xcode, but it is 
version 16.3, well past the 14.3 used for R-4.4.3. Sleuthing some more, 
I realized that it?s an easy mistake to update the 
|/usr/local/lib/libomp.dylib| file to be a different version than what 
is distributed in CRAN?s R. I don?t see an easy way to change version 
within XCode, and when I tried installing XCode-14.3 it says my OS does 
not support that (old) version. (Even with gfortran 12.2, I can?t fix 
that error, but that?s a different subject.)

I recognize the challenge in distributing binaries for multiple versions 
of macos and xcode. I believe R-4.5 is still being built on XCode-14.3, 
which begs another question of if/when XCode updates are planned for R 
binaries and CRAN. (Big topic, I understand, I did not find a recent 
discussion. My apologies if I missed something.)

Thanks,
Bill

​
Simon Urbanek
# 
Bill,

the libomp.dylib inside R is there solely for the CRAN binaries to make sure that package binaries on CRAN can use the correct OpenMP runtime. Is it not intended to be used outside of that setup.

That is also why we don?t ship the headers as anyone compiling from sources will have to do a bit of legwork to make sure it works as it is not entirely trivial to do that correctly. The CRAN build machines are setup such that it "just works" to make it easy on package maintainers. However, there are lot of issues/moving parts when building OpenMP-enabled code yourself (as you found out), so it is not encouraged unless you are very familiar with the issues (correct run-time and complier match-up etc.).

That said, Xcode 16.3 is a big jump in the clang versions (from 17 to 19) so I would not recommend it just yet - have you tried Xcode 16.2? I tried to reproduce your problem and it worked just fine with Xcode 16.2 (CLT 16.2 to be precise and my corresponding OpenMP run-time from LLVM 17.0.6). Xcode 16.2 is supported on all macOS versions (and what the arm64 R builds are moving to) so I would suggest trying that first. (I didn't have CLT 16.3 installed yet, so I?ll have to wait for a quiet minute to see if that?s really what breaks.)

Cheers,
Simon
biii+rsigm@c m@iii@g oii 8p@wexpress@com
# 
Simon,

Long time fan :-)

Downgrading to 16.2 worked. I?m baffled why a /minor/ release change was 
sufficient to make it work, I had assumed that the major different (14 
to 16) would have been more responsible. I guess they aren?t using 
semver? Thank you for that advice, I?m on 16.2 now.

I found 16.2 at https://xcodereleases.com. While the release notes for 
16.3 are detailed (much of it beyond my familiarity), I don?t see 
anything clear that would indicate to me that 16.2 would work and 16.3 
would not. (For instance, no mention of OMP or OpenMP.) Besides you 
having a running familiarity with the topic of R compilation on macos in 
general, is there something that should have clued me in to suggest 
16.2? (Quick background: I?ve been on Linux and FOSS since 1993, when 
floppies were the rage. I?ve used R on Windows and Linux since 2011. I 
started on MacOS and therefore R on MacOS in Dec 2024. I?m definitely a 
freshman on campus here, constantly befuddled by some of the nuanced 
differences in macos.)

    /will have to do a bit of legwork to make sure it works as it is not
    entirely trivial to do that correctly/

My legwork has been educating. Can we ease the burden slightly? The docs 
are clear about XCode 14.2/14.3, but since that version isn?t supported 
on Sequoia there is no hint at a path forward other than to find this 
mailing list and ask (you). The |/openmp/| portion of the website 
discusses downloading one of 26 versions of |openmp-*.tar.gz|; it seems 
risky to me that it would be wise to use a version of OpenMP that is 
different than what R is linked against, ergo /?chaos when more OMP 
run-times get loaded into one process?/.

Perhaps I?m making a hasty assumption here ? the R packaging includes 
libraries for OMP, BLAS, and LAPACK, to name a few, and none of their 
headers are included. I had assumed that including headers for all of 
them would be useful, but I think you?re saying that you?d rather make 
devs learn some of the nuances of building R against those libraries 
before writing packages that link against them, is that right?

How about two alternatives:

 1.

    Update documentation on https://mac.r-project.com to include a note
    about 16.3, perhaps as simple as ?16.3 is known to break some
    packages?. This may be hampered if the ?16.2? downgrade works for
    |data.table| but other packages would require perhaps ?15.*? or
    ?14.*? to compile.

 2.

    Provide a tarball that contains the header files for libraries
    against which a particular version of R is compiled. For instance,
    ?R-headers-4.4.tar.gz? would include the OMP headers and possibly
    (I?m stretching) headers for BLAS and LAPACK. It would be an
    optional download that is updated when the linked libraries are
    updated, probably not at every R release.

    Having a tarball of /just/ headers helps keep the headers the same
    version as the linked |libomp|, and still requires some legwork for
    the devs. I think this encourages safer version compatibility, and
    requires the dev to work harder if they need something further.

Thoughts?

Last oh-btw: you said you are using CLT 16.2, is that available from 
Apple, are you using homebrew to install it, or something else? I?m not 
yet tight on disk space, but I don?t use the rest of XCode so it seems 
I?m wasting some space.

Thanks,
Bill
On 4/22/25 22:31, Simon Urbanek wrote:

            

      
      
      
    

        

      
      
    

        
​

  
  


  


      

    

    

      
      

        


  

        

      Taras Zakharko
    

    

      
      #
    

  


  

      
Bill,

Apple does not support OpenMP in their toolchain, and OpenMP is disabled in XCode. Simon and others rely on the fact that the toolchain is built on top of Clang and thus the OpenMP support can be patched in. However, as far as Apple goes all this is undocumented behavior and subject to change at any time. So I wouldn?t expect things improving any time soon.

Best,

Taras

        

      
      
    

  
  


  


      

    

    

      
      

        


  

        

      biii+rsigm@c m@iii@g oii 8p@wexpress@com
    

    

      
      #
    

  


  

      
Thanks Taras, I understand that, perhaps that portion (semver and big 
changes on minor-looking version bumps) was more of a vent, I should not 
have wasted the email-time on that :-)

As I learn more about the dev required for macos, I?m impressed things 
are a stable as they are, due to strong efforts by Simon and whomever 
else helps on this side of things. It?s also because of this effort and 
fragility that I recommend things like updating docs (for future 
new-to-macos devs) and asking to grease the skids for doing things for 
one?s self (by adding a package of same-version-headers). I think having 
an |R-macos-dev-*.tar.gz| package of headers is consistent with 
many/most OS and application packaging; the biggest question is if there 
are other gotchas I?m not aware of that would make that a moving target. 
(Perhaps it would have to be dependent on more than just the R version ? 
but I thought not since there is one compiled version of R that covers 
most/all macos versions.)

Cheers,
Bill

        
On 4/23/25 11:42, Taras Zakharko wrote:

        

            

      
      
      
    

        

      
      
    

        
​

  
  


  


      

    

    

      
      

        


  

        

      biii+rsigm@c m@iii@g oii 8p@wexpress@com
    

    

      
      #
    

  


  

      
For version diversity, this returns false for me on R-4.4.3 (it?s not a 
4.5 thing).

(If this were linux, I?d strace it to see what?s going on. Since this is 
macos, I cannot get dtrace or execsnoop to get anywhere since SIP is 
enabled.)

Bill

        
On 4/23/25 11:52, bill+rsigmac at 8pawexpress.com wrote:

        

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        
​