Skip to content

[Rcpp-devel] question re: LdFlags, RcppLdFlags

28 messages · Kevin Ushey, Dirk Eddelbuettel, Romain Francois

Original

Messages 1–25 of 28

Kevin Ushey
# 
Hi,

In previous versions of Rcpp, we typically constructed the PKG_LIBS
variable in Makevars through

    PKG_LIBS = `$(R_HOME)/bin/Rscript -e "Rcpp:::LdFlags()"`

This is also what's done in e.g. Rcpp.package.skeleton. However, following
the new guidelines from CRAN, RcppLdFlags was exported, but not LdFlags. I
wonder if the intention was to export LdFlags as well, so that we can
simply omit one colon from the LdFlags call:

    PKG_LIBS = `$(R_HOME)/bin/Rscript -e "Rcpp::LdFlags()"`

As is, because RcppLdFlags was exported and returns its result invisibly,
it's required to write e.g.

    PKG_LIBS = `$(R_HOME)/bin/Rscript -e "cat(Rcpp::RcppLdFlags())"`

which is a bit uglier.

Can we have LdFlags exported in addition to RcppLdFlags? It would also help
keep everything consistent with the available documentation.

-Kevin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.r-forge.r-project.org/pipermail/rcpp-devel/attachments/20131008/9b11ee2b/attachment.html>
Dirk Eddelbuettel
#
On 8 October 2013 at 14:46, Kevin Ushey wrote:
| Hi,
| 
| In previous versions of Rcpp, we typically constructed the PKG_LIBS variable in
| Makevars through
| 
| ? ? PKG_LIBS = `$(R_HOME)/bin/Rscript -e "Rcpp:::LdFlags()"`
| 
| This is also what's done in e.g. Rcpp.package.skeleton. However, following the
| new guidelines from CRAN, RcppLdFlags was exported, but not LdFlags. I wonder

Darn.  Error on my part.  It should have been LdFlags.  That was the intent.
Oh well.

I didn't get bitten by it myself as CRAN as the stupid exception rule "if in
package of some maintainer" or else I would have noticed when preparing
RcppArmadillo. 

| if the intention was to export LdFlags as well, so that we can simply omit one
| colon from the LdFlags call:
| 
| ? ? PKG_LIBS = `$(R_HOME)/bin/Rscript -e "Rcpp::LdFlags()"`
| 
| As is, because RcppLdFlags was exported and returns its result invisibly, it's
| required to write e.g.
| 
| ? ? PKG_LIBS = `$(R_HOME)/bin/Rscript -e "cat(Rcpp::RcppLdFlags())"`
| 
| which is a bit uglier.?

We had variants like that in the earlier which is part of the reason the
particular R file has several such functions.

Only LdFlags() should be used, but until the next release we are stuck with a
typo / think.

| Can we have LdFlags exported in addition to RcppLdFlags? It would also help
| keep everything consistent with the available documentation.

I'll do that right now in SVN.

Thanks for the report.

Dirk
Dirk Eddelbuettel
#
On 8 October 2013 at 16:53, Dirk Eddelbuettel wrote:
| | This is also what's done in e.g. Rcpp.package.skeleton. However, following the
| | new guidelines from CRAN, RcppLdFlags was exported, but not LdFlags. I wonder
| 
| Darn.  Error on my part.  It should have been LdFlags.  That was the intent.
| Oh well.

Minor correction: Seems we need both. So back to export RcppLdFlags(), but
thanks to Kevin also exporting LdFlags().  Done in svn rev4566.

Dirk
Romain Francois
# 
Le 09/10/13 03:41, Dirk Eddelbuettel a ?crit :
One good thing about the previous system was that other packages could 
follow the same convention and also have a hidden LdFlags function 
giving information about what is required to link against it.

I do it, e.g. on the dplyrRcpp package.

Now that it is exported, I get this when I load dplyrRcpp which is 
annoying:


L'objet suivant est masqu? from ?package:Rcpp?:

     LdFlags


What would be the recommendation for such third party packages that also 
ship a library.

Was it really a problem that we called LdFlags with :::, did R CMD check 
look for it in Makevars and Makevars.win ?

It is not really a problem for me, I can eliminate the need for a 
library in dplyrRcpp.

Would it perhaps make sense to think about standard ways for R packages 
to register what they need. We have a system like this for inline 
plugins, maybe it would be interesting to move this up in R and let 
packages communicate in some way (perhaps in DESCRIPTION) what they need.

Romain
Dirk Eddelbuettel
#
On 9 October 2013 at 23:13, Romain Francois wrote:
| Le 09/10/13 03:41, Dirk Eddelbuettel a ?crit :
| >
| > On 8 October 2013 at 16:53, Dirk Eddelbuettel wrote:
| > | | This is also what's done in e.g. Rcpp.package.skeleton. However, following the
| > | | new guidelines from CRAN, RcppLdFlags was exported, but not LdFlags. I wonder
| > |
| > | Darn.  Error on my part.  It should have been LdFlags.  That was the intent.
| > | Oh well.
| >
| > Minor correction: Seems we need both. So back to export RcppLdFlags(), but
| > thanks to Kevin also exporting LdFlags().  Done in svn rev4566.
| >
| > Dirk
| 
| One good thing about the previous system was that other packages could 
| follow the same convention and also have a hidden LdFlags function 
| giving information about what is required to link against it.
| 
| I do it, e.g. on the dplyrRcpp package.
| 
| Now that it is exported, I get this when I load dplyrRcpp which is 
| annoying:
| 
| 
| L'objet suivant est masqu? from ?package:Rcpp?:
| 
|      LdFlags

Gosh, that sucks.  It'll bite a few of mine too.
 
| What would be the recommendation for such third party packages that also 
| ship a library.

Don't know, and am open for suggestions.  

Maybe the local packages need to rename their version?  

Or maybe I got it all wrong and we get by with fewer exports?  I am (as
usual) open for all sorts of changes at the Rcpp level (provided they don't
break anything).

| Was it really a problem that we called LdFlags with :::, did R CMD check 
| look for it in Makevars and Makevars.win ?

CRAN / R Core thought so and told everybody who ran 'R CMD check' with a new
version of R(-devel).  That has been going on for a few weeks.  Not sure if
you regularly build r-devel or run R CMD check, but it's been there for a
good while (and check R's NEWS commit log if you need to know more).

I think R 3.0.2 may now do it too; see NEWS.  

And see Kevin's email from yesterday.  I had also gotten private email from
another maintainer who apparently had CRAN yell at him over this.  <Sigh>

So from the Rcpp side we have little choice but to export LdFlags() as others are
calling it, and R CMD check now complains about the use -- unless it is exported.
 
| It is not really a problem for me, I can eliminate the need for a 
| library in dplyrRcpp.
| 
| Would it perhaps make sense to think about standard ways for R packages 
| to register what they need. We have a system like this for inline 
| plugins, maybe it would be interesting to move this up in R and let 
| packages communicate in some way (perhaps in DESCRIPTION) what they need.

Sounds good, make a proposal, get buy-in from R Core and I'll gladly adapt my
other packages once that is implemented.

Dirk
Romain Francois
# 
Le 09/10/13 23:43, Dirk Eddelbuettel a ?crit :
That will then quickly become a mess. Having the convention 
package:::LdFlags() was fine.

Or perhaps we could centralize within Rcpp, i.e. have something like:

PKG_LIBS = `$(R_HOME)/bin/Rscript -e "Rcpp::LdFlags('foo')"`

and we would just have to leverage the inline plugin for package "foo' 
there.
Sure.
I just meant to start a discussion. I'll adapt my package so that it 
does not need to build a library.
Dirk Eddelbuettel
#
On 10 October 2013 at 00:05, Romain Francois wrote:
| That will then quickly become a mess. Having the convention 
| package:::LdFlags() was fine.

I agree. It was a perfectly legit use of ::: but then the powers of CRAN do
other things do we don't agree with but cannot easily alter.
 
| Or perhaps we could centralize within Rcpp, i.e. have something like:
| 
| PKG_LIBS = `$(R_HOME)/bin/Rscript -e "Rcpp::LdFlags('foo')"`
| 
| and we would just have to leverage the inline plugin for package "foo' 
| there.

That's pretty.  For all the Rcpp-using package, we can assume Rcpp to be
there so it may as well pivot over and call a plugin from 'foo'. 

| > So from the Rcpp side we have little choice but to export LdFlags() as others are
| > calling it, and R CMD check now complains about the use -- unless it is exported.
| 
| Sure.

The other immediate defensive way would be to use NAMESPACE etc on the client
package and not load Rcpp and Rcpp$foo but just importFrom() etc what is needed.
 
Dirk
Kevin Ushey
# 
Hi Dirk, Romain,

For what it's worth, this may be unnecessary for now. There are packages on
CRAN that pass with no notes AFAICS, using Rcpp:::LdFlags in Makevars. Use
of Rcpp:::LdFlags() within Makevars does not trigger any explicit notes for
me in the current R-devel on any of my packages, although a CRAN maintainer
who inspected Makevars might still dislike such use -- I am not sure. I
haven't encountered such a situation yet as I haven't tried submitting any
new packages recently. I'm not sure if any package maintainers have related
that specific frustration off-list. RcppLdFlags being exported was, in
fact, necessary for some other packages that were constructing their
Makevars programmatically through package R code.

In case it's ultimately unnecessary -- sorry for wasting your time on this;
LdFlags can likely remain un-exported.

-Kevin

<snip...>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.r-forge.r-project.org/pipermail/rcpp-devel/attachments/20131009/a498e720/attachment.html>
Dirk Eddelbuettel
#
On 9 October 2013 at 17:14, Dirk Eddelbuettel wrote:
| The other immediate defensive way would be to use NAMESPACE etc on the client
| package and not load Rcpp and Rcpp$foo but just importFrom() etc what is needed.

An 'Imports:' and NAMESPACE entry appears to be all it takes.  RcppGSL also
has its own LdFlags, but with the diff below against what is in svn head I am
passing R CMD check without a hitch on R release and R devel.  

So for dplyrRcpp, if you just want to suppress the warning, simply alter your
DESCRIPTION and NAMESPACE accordingly.

Dirk


edd at max:~/svn/rcpp/pkg$ svn di RcppGSL
Index: RcppGSL/DESCRIPTION
===================================================================
--- RcppGSL/DESCRIPTION	(revision 4542)
+++ RcppGSL/DESCRIPTION	(working copy)
@@ -1,7 +1,7 @@
 Package: RcppGSL
 Type: Package
 Title: Rcpp integration for GNU GSL vectors and matrices
-Version: 0.2.0.2
+Version: 0.2.0.3
 Date: $Date$
 Author: Romain Francois and Dirk Eddelbuettel
 Maintainer: Dirk Eddelbuettel <edd at debian.org>
@@ -27,8 +27,8 @@
   write a similar package against another library.
 License: GPL (>= 2)
 LazyLoad: yes
-Imports: Rcpp (>= 0.9.8)
 Suggests: RUnit, inline, highlight
 LinkingTo: Rcpp
+Imports: Rcpp
 SystemRequirements: GNU GSL
 VignetteBuilder: highlight
Index: RcppGSL/NAMESPACE
===================================================================
--- RcppGSL/NAMESPACE	(revision 4542)
+++ RcppGSL/NAMESPACE	(working copy)
@@ -1,6 +1,7 @@
 useDynLib(RcppGSL)
 
 importFrom(utils,assignInNamespace)
+importFrom(Rcpp,LdFlags)
 export(fastLmPure,
        fastLm)
 S3method(fastLm, default)
edd at max:~/svn/rcpp/pkg$
Dirk Eddelbuettel
# 
Kevin,
On 9 October 2013 at 15:37, Kevin Ushey wrote:
| For what it's worth, this may be unnecessary for now. There are packages on
| CRAN that pass with no notes AFAICS, using Rcpp:::LdFlags in Makevars.?Use of
| Rcpp:::LdFlags() within Makevars does not trigger any explicit notes for me in
| the current R-devel on any of my packages, although a CRAN maintainer who
| inspected Makevars might still dislike such use -- I am not sure. I haven't
| encountered such a situation yet as I haven't tried submitting any new packages
| recently. I'm not sure if any package maintainers have related that specific
| frustration off-list. RcppLdFlags being exported was, in fact, necessary for
| some other packages that were constructing their Makevars programmatically
| through package R code.
| 
| In case it's ultimately unnecessary -- sorry for wasting your time on this;
| LdFlags can likely remain un-exported.

Maybe.  But to move in either direction we'd need to go from 'maybe' and
'might' and 'coulda/shoulda' to something with more certainty.

The warnings certainly hit at some point. But as I recall CRAN / R Core also
rolled some parts of the ':::' testing back.  I also recall that they talked
about some extra nonsense of allowing ':::' between packages with the same
maintainer (which efffectively prevents from testing on my package, sigh) 

Maybe someone could volunteer and tests a few cases?

In any event there is no immediate release ... so we can alter NAMESPACE for
Rcpp as well a few more times til we feel it is ready and good.

Dirk
Romain Francois
# 
Le 10/10/13 00:43, Dirk Eddelbuettel a ?crit :
RcppGSL does not export its LdFlags, if it did, you'd see the warning 
when you load it.

So we would use Rcpp::LdFlags and RcppGSL:::LdFlags. <nonsense/>



Why would you import Rcpp's LdFlags inside RcppGSL's namespace, you 
don't use it there.
Does not cut it sorry.

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Romain Francois
    

    

      
      #
    

  


  

      
Le 10/10/13 00:14, Dirk Eddelbuettel a ?crit :

            

      
      
      
    

        
I think it is worth looking in that direction.

            

      
      
      
    

        
Maybe that works.

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Dirk Eddelbuettel
    

    

      
      #
    

  


  

      
(Combining two of your emails)

        
On 10 October 2013 at 01:08, Romain Francois wrote:

        
| RcppGSL does not export its LdFlags, if it did, you'd see the warning 
| when you load it.

Good catch ... but turns out I don't (to my surprise).

It seems to work. Maybe NAMESPACE really do help.  Full log at bottom.
 
| Why would you import Rcpp's LdFlags inside RcppGSL's namespace, you 
| don't use it there.

R CMD check did complain, I guess it must grep for src/ to find it in
Makevars. Once I imported the complaint went away.

        
On 10 October 2013 at 01:18, Romain Francois wrote:

        
| > That's pretty.  For all the Rcpp-using package, we can assume Rcpp to be
| > there so it may as well pivot over and call a plugin from 'foo'.
| 
| I think it is worth looking in that direction.

We'll see if we need it. Kevin's point etc.
 
| > The other immediate defensive way would be to use NAMESPACE etc on the client
| > package and not load Rcpp and Rcpp$foo but just importFrom() etc what is needed.
| 
| Maybe that works.

Generally speaking, and as I understand recent r-devel threads, Imports: and
importFrom(...) is how we are supposed to work with NAMESPACEs now.  That is
also what I did here, and what you roundly mocked one email ago (see above).

I am sure we'll find a way to sort it all out. 

Dirk

Log below

edd at max:~/svn/rcpp/pkg$ svn di RcppGSL     ## now exporting LdFlags in RcppGSL
Index: RcppGSL/NAMESPACE
===================================================================
--- RcppGSL/NAMESPACE	(revision 4567)
+++ RcppGSL/NAMESPACE	(working copy)
@@ -3,7 +3,8 @@
 importFrom(utils,assignInNamespace)
 importFrom(Rcpp,LdFlags)
 export(fastLmPure,
-       fastLm)
+       fastLm,
+       LdFlags)
 S3method(fastLm, default)
 S3method(fastLm, formula)
 S3method(predict, fastLm)
edd at max:~/svn/rcpp/pkg$ R CMD INSTALL RcppGSL_0.2.0.3.tar.gz ## using rebuilt tar.gz
* installing to library ?/usr/local/lib/R/site-library?
* installing *source* package ?RcppGSL? ...
checking for gcc... gcc
checking for C compiler default output file name... a.out
checking whether the C compiler works... yes
checking whether we are cross compiling... no
checking for suffix of executables... 
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking for gsl-config... /usr/bin/gsl-config
checking for Rscript... yes
configure: creating ./config.status
config.status: creating src/Makevars
** libs
ccache g++-4.7 -I/usr/share/R/include -DNDEBUG -I/usr/include -I../inst/include  -I"/usr/local/lib/R/site-library/Rcpp/include"   -fpic  -g -O3 -Wall -pipe -Wno-unused -pedantic  -c fastLm.cpp -o fastLm.o
g++-4.7 -shared -o RcppGSL.so fastLm.o -L/usr/lib -lgsl -lgslcblas -lm -L/usr/local/lib/R/site-library/Rcpp/lib -lRcpp -Wl,-rpath,/usr/local/lib/R/site-library/Rcpp/lib -L/usr/lib/R/lib -lR
installing to /usr/local/lib/R/site-library/RcppGSL/libs
** R
** inst
** preparing package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded
* DONE (RcppGSL)
edd at max:~/svn/rcpp/pkg$ R

R version 3.0.2 (2013-09-25) -- "Frisbee Sailing"
Copyright (C) 2013 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.


Careful now, or I'll get into a rant about how much you pay for ArcGIS and whether you are getting value for money if bugs become 'well-known'! Bugs in R
tend to be either 'obscure' or 'fixed' :)
   -- Barry Rowlingson (in response to a request for rgdal to work around a well-known bug in ArcGIS)
      R-SIG-Geo (March 2012)

R> library(RcppGSL)          ## no warning !!
R> LdFlags                   ## local 
function (print = TRUE) 
{
    if (print) 
        cat(.pkgglobalenv$gsl_libs)
    else .pkgglobalenv$gsl_libs
}
<environment: namespace:RcppGSL>
R> Rcpp::LdFlags             ## remote from Rcpp
function (static = staticLinking()) 
{
    cat(RcppLdFlags(static = static))
}
<environment: namespace:Rcpp>
R>

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Romain Francois
    

    

      
      #
    

  


  

      
Le 10/10/13 02:41, Dirk Eddelbuettel a ?crit :

            

      
      
      
    

        
Yes; My mistake. almost 3 am here :/

            

      
      
      
    

        
If this fixes something for you, that's fine. I just don't get why you 
would import something you don't use.

What makes sense to me now is both Rcpp and RcppGSL exporting their own 
LdFlags. Having RcppGSL import the one from Rcpp makes no sense to me. 
But if it makes R CMD check happy.

Soon they are going to rename it R CMD ChuckNorris

            

      
      
      
    

        
I'm sorry you preceived mockery.

            

      
      
      
    

        

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Dirk Eddelbuettel
    

    

      
      #
    

  


  

      
On 10 October 2013 at 02:56, Romain Francois wrote:

        
| Le 10/10/13 02:41, Dirk Eddelbuettel a ?crit :
| >
| > (Combining two of your emails)
| >

        
| > On 10 October 2013 at 01:08, Romain Francois wrote:

        
| > | RcppGSL does not export its LdFlags, if it did, you'd see the warning
| > | when you load it.
| >
| > Good catch ... but turns out I don't (to my surprise).
| 
| Yes; My mistake. almost 3 am here :/

Understood.  

But also -- "this may yet be different": For RcppGSL, CFlags() and LdFlags()
export the -I... and -L... -l... values for _the GNU GSL_ and not for us. I
think I once wrote that to help package mvabund which of course still doesn't
use it (and still uses a bunch of should-be-replaced fragments, but heck, not
my package).

So nothing actually used this until ... I got RcppZiggurat version 0.0.1 onto
CRAN on Sunday.  So I can say with a straight face that the functions are
used on CRAN (and made linking to the GSL a total breeze, incl Windows builds
via win-builder etc).

| > | Why would you import Rcpp's LdFlags inside RcppGSL's namespace, you
| > | don't use it there.
| >
| > R CMD check did complain, I guess it must grep for src/ to find it in
| > Makevars. Once I imported the complaint went away.
| 
| If this fixes something for you, that's fine. I just don't get why you 
| would import something you don't use.

To make R CMD check happy, mostly.  As I said in this thread. I was quite
fine with how we use ':::' here.  "They were not."  So I cope and adapt.
 
| What makes sense to me now is both Rcpp and RcppGSL exporting their own 
| LdFlags. Having RcppGSL import the one from Rcpp makes no sense to me. 
| But if it makes R CMD check happy.

It's all pretty odd, see above.

Now: does dplyrRcpp actually export its library?  Who would link against it?
 
| Soon they are going to rename it R CMD ChuckNorris

Saw your tweet. Quite.
 
| > Generally speaking, and as I understand recent r-devel threads, Imports: and
| > importFrom(...) is how we are supposed to work with NAMESPACEs now.  That is
| > also what I did here, and what you roundly mocked one email ago (see above).
| 
| I'm sorry you preceived mockery.

Well, that's how your tone comes across a lot lately over on my end. But then
neither one of us is a native speaker in the language we use to communicate,
so their may be some loss in the signal.

Dirk

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Romain Francois
    

    

      
      #
    

  


  

      
Le 10/10/13 03:21, Dirk Eddelbuettel a ?crit :

            

      
      
      
    

        
Yes.

Whoever who wants to use the classes it defines. dplyrRcpp exists mainly 
for facilitating the implementation of dplyr, but some of the concepts 
are general and could be used as building blocks of other things.

I could/will make these things as header only, or using GetCCallable 
anyway so linking is not a long term problem.

            

      
      
      
    

        
That was tweet worthy

            

      
      
      
    

        

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Dirk Eddelbuettel
    

    

      
      #
    

  


  

      
On 10 October 2013 at 03:29, Romain Francois wrote:

        
| Whoever who wants to use the classes it defines. dplyrRcpp exists mainly 
| for facilitating the implementation of dplyr, but some of the concepts 
| are general and could be used as building blocks of other things.
| 
| I could/will make these things as header only, or using GetCCallable 
| anyway so linking is not a long term problem.

Ack.  If the set of functions is small and rarely changes, then registration
using R's mechanism is indeed fine. Worked so far for getting Jeff and Josh's
xts into RcppXts, but that hasn't really moved beyond proof of concept.

Dirk

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Romain Francois
    

    

      
      #
    

  


  

      
Le 10/10/13 03:53, Dirk Eddelbuettel a ?crit :

            

      
      
      
    

        
That is what Rcpp11 uses too and the unit test suite is big. This works.

Also used in Matrix, lme4.

That's a bit more than POC level.

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Dirk Eddelbuettel
    

    

      
      #
    

  


  

      
On 10 October 2013 at 04:02, Romain Francois wrote:

        
| Le 10/10/13 03:53, Dirk Eddelbuettel a ?crit :
| > Ack.  If the set of functions is small and rarely changes, then registration
| > using R's mechanism is indeed fine. Worked so far for getting Jeff and Josh's
| > xts into RcppXts, but that hasn't really moved beyond proof of concept.

"proof of concept ... at the level of the RcppXts package" in the sense that
the package does not yet do much with the functions it imports.

| That is what Rcpp11 uses too and the unit test suite is big. This works.
| 
| Also used in Matrix, lme4.
| 
| That's a bit more than POC level.

Of course, I phrased this poorly and you promptly misunderstood. 

Registration of C routines has been documented well and used by lme4 and
Matrix for several years now.

But still a pain to declare 'by hand' for a bazillion functions.  I know you
luuv your macros, but you also know how a large part of the world thinks
about old-C-style macros.  But they get certain jobs done. 

Dirk

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Romain Francois
    

    

      
      #
    

  


  

      
Le 10/10/13 04:54, Dirk Eddelbuettel a ?crit :

            

      
      
      
    

        
I don't particularly love macros. I much prefer templates. For example 
in Rcpp11, I replaced thousands lines worth of macros code bloat with a 
few dozens of lines of templates.

Would be easy enough to come up with a way to automatically generate 
things automatically.

A package author could leverage tools:::read_symbols_from_object_file, 
i.e. here are all the symbols in Rcpp.so:

symbols <- tools:::read_symbols_from_object_file( system.file( "libs", 
"Rcpp.so", package = "Rcpp" ) )[,1]

Would be easy to then come up with a naming convention, and generate the 
registration.

read_symbols_from_object_file is why we have been all of a sudden 
forbidden to use functions that R exposes but "not part of the api".

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Dirk Eddelbuettel
    

    

      
      #
    

  


  

      
To bring a bit of closure to this endless thread: It appear from his commit
logs that Romain made the change in his dplyRcpp project we are now supposed
to make per R Core: Demote packages from Depends: to Imports:, and import
what is needed.  

That, as best as I can tell, seems to be the way to avoid the clashes seen in
Romain's first email in this thread.

Some more questions re some possible future use below

        
On 10 October 2013 at 05:15, Romain Francois wrote:

        
| I don't particularly love macros. I much prefer templates. For example 
| in Rcpp11, I replaced thousands lines worth of macros code bloat with a 
| few dozens of lines of templates.

Good to see Romain-the-template-whiz clean up after Romain-the-macro-user. :)
And obviously I don't mind all the macros as your use of these macros gives
us all thse awesome features in Rcpp. But macros are a last resort...

| Would be easy enough to come up with a way to automatically generate 
| things automatically.
| 
| A package author could leverage tools:::read_symbols_from_object_file, 
| i.e. here are all the symbols in Rcpp.so:
| 
| symbols <- tools:::read_symbols_from_object_file( system.file( "libs", 
| "Rcpp.so", package = "Rcpp" ) )[,1]
| 
| Would be easy to then come up with a naming convention, and generate the 
| registration.
| 
| read_symbols_from_object_file is why we have been all of a sudden 
| forbidden to use functions that R exposes but "not part of the api".

Can you (portably, with all (major) compilers and OS choices) pick out
symbols even when they have attribute_hidden set as R does all over its
sources?

Dirk

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Romain Francois
    

    

      
      #
    

  


  

      
Le 10/10/13 16:51, Dirk Eddelbuettel a ?crit :

            

      
      
      
    

        
Sure. I'm counting 515 macros in Rcpp:

% grep "#define" **/* * 2> /dev/null | grep -v "_h" | grep -v "_H" | wc -l
      515

And only 137 in Rcpp11

% grep "#define" **/* 2> /dev/null | grep -v "_h" | grep -v "_H" | wc -l
      137

Macros are evil for things like this:

#define CONSTANT 42
#define max(x,y) (x>y):x?y ;

which can be replaced by const int and inline function. This is 
Effective C++ item #2.

They however can have a role to play in code generation. Here is one 
from Rcpp11:

#define DOT_CALL(name) DotCall(#name, &name)

So that one can write:

DOT_CALL(foo)

instead of DotCall("foo", foo)


They are quite useful for debugging too:

#if RCPP_DEBUG_LEVEL > 0
#define RCPP_DEBUG( fmt, ... ) Rprintf( "%20s:%4d             " fmt "\n" 
, short_file_name(__FILE__), __LINE__, ##__VA_ARGS__ ) ;
#else
#define RCPP_DEBUG( MSG, ... )
#endif

So that we can write:

RCPP_DEBUG( "bla bla %d", 42 )

and leave it there when the code is not debugged.

            

      
      
      
    

        
No.

But since that sort of thing are typically the task of the package 
maintainer: I don't care.

read_symbols_from_object_file is just a convoluted way to call nm.

  tools:::read_symbols_from_object_file
function (f)
{
     if (!nzchar(nm <- Sys.which("nm")))
         return()
     f <- file_path_as_absolute(f)
     if (!(file.info(f)$size))
         return()
     s <- strsplit(system(sprintf("%s -Pg %s", shQuote(nm), shQuote(f)),
         intern = TRUE), " +")
     n <- length(s)
     tab <- matrix("", nrow = n, ncol = 4L)
     colnames(tab) <- c("name", "type", "value", "size")
     i <- rep.int(seq_len(n), sapply(s, length))
     j <- unlist(lapply(s, seq_along))
     tab[n * (j - 1L) + i] <- unlist(s)
     tab
}

on windows, there is an equivalent function that does the windows thing.

But again, I don't care. Generating code is the task of the package 
maintainer, so if I want to use something like that, it just has to work 
on whatever I use.

That's similar to calling compileAttributes, or generating the code 
bloat we've put in Rcpp with the generator scripts.

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Dirk Eddelbuettel
    

    

      
      #
    

  


  

      
On 10 October 2013 at 17:15, Romain Francois wrote:

        
| They are quite useful for debugging too:
| 
| #if RCPP_DEBUG_LEVEL > 0
| #define RCPP_DEBUG( fmt, ... ) Rprintf( "%20s:%4d             " fmt "\n" 
| , short_file_name(__FILE__), __LINE__, ##__VA_ARGS__ ) ;
| #else
| #define RCPP_DEBUG( MSG, ... )
| #endif
| 
| So that we can write:
| 
| RCPP_DEBUG( "bla bla %d", 42 )
| 
| and leave it there when the code is not debugged.

Thanks for all the details and counts on macros. I was mostly aware of this
as you may recall that I wrote a few of those variants in Rcpp too, including
some of the DEBUG ones.

| > Can you (portably, with all (major) compilers and OS choices) pick out
| > symbols even when they have attribute_hidden set as R does all over its
| > sources?
| 
| No.
| 
| But since that sort of thing are typically the task of the package 
| maintainer: I don't care.

I don't follow at all. You appeared to be saying you had a solution to the
linking issue, I point out that hidden symbols are still hidden, and you say
it doesn't matter?  How so?
 
| read_symbols_from_object_file is just a convoluted way to call nm.
| 
|   tools:::read_symbols_from_object_file
| function (f)
| {
|      if (!nzchar(nm <- Sys.which("nm")))
|          return()
|      f <- file_path_as_absolute(f)
|      if (!(file.info(f)$size))
|          return()
|      s <- strsplit(system(sprintf("%s -Pg %s", shQuote(nm), shQuote(f)),
|          intern = TRUE), " +")
|      n <- length(s)
|      tab <- matrix("", nrow = n, ncol = 4L)
|      colnames(tab) <- c("name", "type", "value", "size")
|      i <- rep.int(seq_len(n), sapply(s, length))
|      j <- unlist(lapply(s, seq_along))
|      tab[n * (j - 1L) + i] <- unlist(s)
|      tab
| }

That is something different from the linking issue we were discussing. This
is a pretty printer for (non-stripped) object code using nm(1).
 
| But again, I don't care. Generating code is the task of the package 
| maintainer, so if I want to use something like that, it just has to work 
| on whatever I use.

I believe we were talking about two somewhat distinct issues here.  

Dirk

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Romain Francois
    

    

      
      #
    

  


  

      
Le 10/10/13 17:36, Dirk Eddelbuettel a ?crit :

            

      
      
      
    

        
This is not private email. You were rambling about overuse of macros, I 
was just trying to show that :
- sometimes they do something that can't be done
- I'm using less of them in Rcpp11, so I'm definitely concerned about 
macro overuse.

            

      
      
      
    

        
Nevermind.

            

      
      
      
    

        
Of course hidden symbols are still hidden. what's the point of hiding 
them if you can see them anyway.

To give some context, I'm taking about using R's linking mechanism. I'm 
talking about generating automatically the code that:
- registers a function with R_RegisterCCallable
- grabs the function with R_GetCCallable

read_symbols_from_object_file gives me the names of all the objects in a 
package .so file, then I can grep around for some pattern and do some 
code generation.

I'm suggesting that as a possible way to not have to write the code by 
hand as you seem to not want to do that.

We don't really need read_symbols_from_object_file for that, we could 
have something like an [[Rcpp::register]] attribute so that when we 
write this in a package .cpp:

// [[Rcpp::register]]
void foo( Bar boom ){
    // whatever
}

Using something like read_symbols_from_object_file would recognize these 
funcions based on their names, that can be:
- using some known pattern, e.g.

void foo__impl()

- the fact that the function is inside some known namespace, e.g.

namespace Export {
     void foo() ;
}

The name as given by nm definitely contains namespace information.

            

      
      
      
    

        
But not unrelated.

            

      
      
      
    

        
I see a connection.

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Dirk Eddelbuettel
    

    

      
      #
    

  


  

      
On 10 October 2013 at 18:02, Romain Francois wrote:

        
| To give some context, I'm taking about using R's linking mechanism. I'm 
| talking about generating automatically the code that:
| - registers a function with R_RegisterCCallable
| - grabs the function with R_GetCCallable

[...]

| The name as given by nm definitely contains namespace information.

You are making the heroic assumption that libraries are never stripped, this
is something you have no control.  Eg each of the 100+ r-cran-* packages in
Debian and Ubuntu will by default be stripped, leading to your approach to go
belly-up: 

edd at max:~$ nm /usr/lib/R/site-library/tseries/libs/tseries.so 
nm: /usr/lib/R/site-library/tseries/libs/tseries.so: no symbols
edd at max:~$ 

"No symbols".

This is a standard distro package a user may have:

edd at max:~$ dpkg -S /usr/lib/R/site-library/tseries/libs/tseries.so
r-cran-tseries: /usr/lib/R/site-library/tseries/libs/tseries.so
edd at max:~$ 


So this approach can't be assumed to work _portably across compilers and os
[distros]_ as I asked about earlier -- and which you chose to ignore.

You can look at nm _on your box_ to learn about (unstripped) libraries.  But
I would surmise that you cannot rely on this to build a tool chain.

Dirk