I am building an R package (https://github.com/gdkrmr/zarr-R) that wraps a couple of C++ headers and the resulting shared object is huge (>20Mb, see https://travis-ci.org/gdkrmr/zarr-R). I have tried to set the following in `./src/Makevars`: PKG_CPPFLAGS=-I"../inst/include" -ffunction-sections -fdata-sections PKG_LIBS=-lstdc++fs -flto -Wl,--gc-sections but this does not reduce the size of the shared object and produces some warnings in `R CMD check` due to portability issues. Is there a way to reduce the size of the shared object or do I just have to live with it? Cheers, Guido
[R-pkg-devel] reduce size of shared objects
6 messages · Dirk Eddelbuettel, Guido Kraemer, Iñaki Ucar
On Wed, 11 Dec 2019 at 14:07, Guido Kraemer <gkraemer at bgc-jena.mpg.de> wrote:
I am building an R package (https://github.com/gdkrmr/zarr-R) that wraps a couple of C++ headers and the resulting shared object is huge (>20Mb, see https://travis-ci.org/gdkrmr/zarr-R). I have tried to set the following in `./src/Makevars`: PKG_CPPFLAGS=-I"../inst/include" -ffunction-sections -fdata-sections PKG_LIBS=-lstdc++fs -flto -Wl,--gc-sections but this does not reduce the size of the shared object and produces some warnings in `R CMD check` due to portability issues. Is there a way to reduce the size of the shared object or do I just have to live with it?
Hi, see Dirk's http://dirk.eddelbuettel.com/blog/2019/03/27/ and related posts (linked in the first paragraph). I?aki
On 11 December 2019 at 14:07, Guido Kraemer wrote:
| I am building an R package (https://github.com/gdkrmr/zarr-R) that wraps | a couple of C++ headers and the resulting shared object is huge (>20Mb, | see https://travis-ci.org/gdkrmr/zarr-R). I have tried to set the | following in `./src/Makevars`: | | PKG_CPPFLAGS=-I"../inst/include" -ffunction-sections -fdata-sections | PKG_LIBS=-lstdc++fs -flto -Wl,--gc-sections | | but this does not reduce the size of the shared object and produces some | warnings in `R CMD check` due to portability issues. Is there a way to | reduce the size of the shared object or do I just have to live with it? Yes, a known issue with C++. I blogged about it a few times http://dirk.eddelbuettel.com/blog/2019/03/27#021_stripping_take_three http://dirk.eddelbuettel.com/blog/2017/08/20#010_stripping_shared_libraries http://dirk.eddelbuettel.com/blog/2017/08/14#009_compact_shared_libraries But CRAN does not allow stripping; just this week I updated a package at their request removing a strip invocation. So I think you can only add the instructions _locally_ in ~/.R/Makevars -- I have this STRIP=-Wl,-S SHLIB_CXXLDFLAGS = $(STRIP) -shared SHLIB_CXX11LDFLAGS = $(STRIP) -shared SHLIB_CXX14LDFLAGS = $(STRIP) -shared SHLIB_FCLDFLAGS = $(STRIP) -shared SHLIB_LDFLAGS = $(STRIP) -shared but not in shipped packages going e.g. to CRAN. Dirk
http://dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org
Thanks, I wasn't aware about these blog posts. I have just read a little bit of gcc documentation and the `-flto` flag has to be provided also in the compile steps, doing this PKG_CPPFLAGS=-I"../inst/include" -flto PKG_LIBS=-lstdc++fs -flto reduces the size of the final `.so` to under 1Mb, but gives a warning in `R CMD check --as-cran` because `-flto` doesn't seem to be portable. Cheers, Guido
On 12/11/19 2:15 PM, I?aki Ucar wrote:
On Wed, 11 Dec 2019 at 14:07, Guido Kraemer <gkraemer at bgc-jena.mpg.de> wrote:
I am building an R package (https://github.com/gdkrmr/zarr-R) that wraps a couple of C++ headers and the resulting shared object is huge (>20Mb, see https://travis-ci.org/gdkrmr/zarr-R). I have tried to set the following in `./src/Makevars`: PKG_CPPFLAGS=-I"../inst/include" -ffunction-sections -fdata-sections PKG_LIBS=-lstdc++fs -flto -Wl,--gc-sections but this does not reduce the size of the shared object and produces some warnings in `R CMD check` due to portability issues. Is there a way to reduce the size of the shared object or do I just have to live with it?
Hi, see Dirk's http://dirk.eddelbuettel.com/blog/2019/03/27/ and related posts (linked in the first paragraph). I?aki
Guido Kraemer Max Planck Institute for Biogeochemistry Jena Department for Biogeochemical Integration Hans-Kn?ll-Str. 10 07745 Jena Germany phone: +49 3641 576293 e-mail: gkraemer at bgc-jena.mpg.de
I have tried the suggestion from the blog post and added |strippedLib: $(SHLIB) if test -e "/usr/bin/strip" & test -e "/bin/uname" & [[ `uname` == "Linux" ]] ; \ then /usr/bin/strip --strip-debug $(SHLIB); fi .phony: strippedLib| || || |to `src/Makevars` and it works great, it gets the size of the shared object down to 1.3 Mb.| |When submitting this to CRAN some day, should I apply this trick or ignore it and let the user| |decide with the new ``R CMD install --strip`` option in R 3.6? Cheers, Guido | || | |
On 12/11/19 2:25 PM, Guido Kraemer wrote:
Thanks, I wasn't aware about these blog posts. I have just read a little bit of gcc documentation and the `-flto` flag has to be provided also in the compile steps, doing this PKG_CPPFLAGS=-I"../inst/include" -flto PKG_LIBS=-lstdc++fs -flto reduces the size of the final `.so` to under 1Mb, but gives a warning in `R CMD check --as-cran` because `-flto` doesn't seem to be portable. Cheers, Guido On 12/11/19 2:15 PM, I?aki Ucar wrote:
On Wed, 11 Dec 2019 at 14:07, Guido Kraemer <gkraemer at bgc-jena.mpg.de> wrote:
I am building an R package (https://github.com/gdkrmr/zarr-R) that wraps a couple of C++ headers and the resulting shared object is huge (>20Mb, see https://travis-ci.org/gdkrmr/zarr-R). I have tried to set the following in `./src/Makevars`: PKG_CPPFLAGS=-I"../inst/include" -ffunction-sections -fdata-sections PKG_LIBS=-lstdc++fs -flto -Wl,--gc-sections but this does not reduce the size of the shared object and produces some warnings in `R CMD check` due to portability issues. Is there a way to reduce the size of the shared object or do I just have to live with it?
Hi, see Dirk's http://dirk.eddelbuettel.com/blog/2019/03/27/ and related posts (linked in the first paragraph). I?aki
Guido Kraemer Max Planck Institute for Biogeochemistry Jena Department for Biogeochemical Integration Hans-Kn?ll-Str. 10 07745 Jena Germany phone: +49 3641 576293 e-mail: gkraemer at bgc-jena.mpg.de
On Wed, 11 Dec 2019 at 15:01, Guido Kraemer <gkraemer at bgc-jena.mpg.de> wrote:
I have tried the suggestion from the blog post and added |strippedLib: $(SHLIB) if test -e "/usr/bin/strip" & test -e "/bin/uname" & [[ `uname` == "Linux" ]] ; \ then /usr/bin/strip --strip-debug $(SHLIB); fi .phony: strippedLib| || || |to `src/Makevars` and it works great, it gets the size of the shared object down to 1.3 Mb.| |When submitting this to CRAN some day, should I apply this trick or ignore it and let the user| |decide with the new ``R CMD install --strip`` option in R 3.6? Cheers, Guido |
Not sure if you received Dirk's reponse: https://stat.ethz.ch/pipermail/r-package-devel/2019q4/004763.html
From his experience, it seems that CRAN does not allow stripping, so
follow his intructions to strip objects locally using your ~/.R/Makevars instead of the package's src/Makevars. I?aki