[Rcpp-devel] package using Rcpp aborting with: terminate called after throwing an instance of 'Rcpp::not_compatible' inside class__newInstance()

One package (guitar) that happens to use Rcpp started breaking recently without any change to its own code as follows:
 install.packages("guitar",,"http://rforge.net")
 library(guitar)
 Repository$new(".")
terminate called after throwing an instance of 'Rcpp::not_compatible'
  what():  expecting an external pointer
Aborted (core dumped)

Stack trace below - it is happening in Module.cpp class__newInstance(). The method being called is simply a static constructor:

class Repository: public CPPWrapperObjectTraits<Repository, git_repository>
{
public:
    explicit Repository(std::string path);

First, I wouldn't expect an abort - ideally if anything is wrong it should just raise an R error (aborts are much harder to trace - in particular since the original issue was deep in a chain of several packages so R traceback would be helpful). Second, XPtr gets called with R_NilValue inside class__newInstance() but I have no idea why since the Repository object looks legit:
Repository
C++ class 'Repository' <0x7f934961be60>
Constructors:
    Repository(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)

Fields: No public fields exposed by this class

Methods: 
     SEXP commits(SEXP)  

[...]

Note that it aborts even before the constructor is called so no code from the Repository object is actually involved here - it's inside Rcpp.

I have inherited the code (so please no flames, I just have to fix it ;)), so any leads as to why it now breaks while it used to work just fine would be highly appreciated. The sources are at
https://github.com/s-u/guitar
and the above fails both on Ubuntu and OS X (tested with R 3.2.2 and Rcpp 0.12.1 and 0.12.2 respectively).

Thanks,
Simon

stack trace (based on Rcpp 0.12.2) and I have instrumented XPtr - the R object passed is simply R_NilValue:

libc++abi.dylib: terminating with uncaught exception of type Rcpp::not_compatible: expecting an external pointer
Process 8286 stopped
* thread #1: tid = 0x155af, 0x00007fff8cdf5866 libsystem_kernel.dylib`__pthread_kill + 10, queue = 'com.apple.main-thread', stop reason = signal SIGABRT
    frame #0: 0x00007fff8cdf5866 libsystem_kernel.dylib`__pthread_kill + 10
libsystem_kernel.dylib`__pthread_kill + 10:
-> 0x7fff8cdf5866:  jae    0x7fff8cdf5870            ; __pthread_kill + 20
   0x7fff8cdf5868:  movq   %rax, %rdi
   0x7fff8cdf586b:  jmp    0x7fff8cdf2175            ; cerror_nocancel
   0x7fff8cdf5870:  retq   
(lldb) bt
* thread #1: tid = 0x155af, 0x00007fff8cdf5866 libsystem_kernel.dylib`__pthread_kill + 10, queue = 'com.apple.main-thread', stop reason = signal SIGABRT
  * frame #0: 0x00007fff8cdf5866 libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fff94a0a35c libsystem_pthread.dylib`pthread_kill + 92
    frame #2: 0x00007fff8e171b1a libsystem_c.dylib`abort + 125
    frame #3: 0x00007fff95267f31 libc++abi.dylib`abort_message + 257
    frame #4: 0x00007fff9528d93a libc++abi.dylib`default_terminate_handler() + 240
    frame #5: 0x00007fff95c28322 libobjc.A.dylib`_objc_terminate() + 124
    frame #6: 0x00007fff9528b1d1 libc++abi.dylib`std::__terminate(void (*)()) + 8
    frame #7: 0x00007fff9528ac5b libc++abi.dylib`__cxa_throw + 124
    frame #8: 0x00000001044d86df Rcpp.so`Rcpp::XPtr<Rcpp::Module, Rcpp::PreserveStorage, &(this=<unavailable>, x=<unavailable>, tag=<unavailable>, prot=<unavailable>))>::XPtr(SEXPREC*, SEXPREC*, SEXPREC*) + 319 at XPtr.h:67
    frame #9: 0x00000001044d67ed Rcpp.so`class__newInstance(SEXPREC*) [inlined] Rcpp::XPtr<Rcpp::Module, Rcpp::PreserveStorage, &(this=0x0000000100806378, x=<unavailable>, tag=<unavailable>, prot=<unavailable>))>::XPtr(SEXPREC*, SEXPREC*, SEXPREC*) + 77 at XPtr.h:71
    frame #10: 0x00000001044d67e2 Rcpp.so`class__newInstance(args=<unavailable>) + 66 at Module.cpp:140
    frame #11: 0x000000010007406a libR.dylib`do_External(call=<unavailable>, op=<unavailable>, args=<unavailable>, env=<unavailable>) + 378 at dotcode.c:548
[...]
Simon,

That may be a little tricky for a few reasons:

 i)   We just released Rcpp 0.12.2. I spend a considerable amount of time
      testing prior to a release. This time we had three full reverse
      dependency checks totalling over 1500 builds given the 500+ revdeps.
      Any such issue would have stopped us. 

 ii)  AFAIK Carlos wrote guitar years ago yet he never pushed it to CRAN. We
      made changes to Rcpp Modules during this time, we also made changes to
      how exception escalate (different OS drive this...).  Because this
      package only ever stayed on GitHub this did not get tested.  Else we
      would have known sooner.

 iii) Rcpp Modules is a clever trick that is rather useful. Sadly, it is not
      that well documented and its principal creator has moved on too so the
      code is not exactly actively maintained.

So here we need a volunteers to dive it this and debug, maybe bisect with
older Rcpp releases, ...

It looks like this won't be Carlos, and it probably won't be me either. 

Dirk
http://dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org
Hi, Simon,

Is there any specific reason to use this?

If you need a binding for libgit2, maybe you can try
https://github.com/ropensci/git2r

At least it is actively maintained.

Best,

KK

Simon,

That may be a little tricky for a few reasons:

 i)   We just released Rcpp 0.12.2. I spend a considerable amount of time
      testing prior to a release. This time we had three full reverse
      dependency checks totalling over 1500 builds given the 500+ revdeps.
      Any such issue would have stopped us.

 ii)  AFAIK Carlos wrote guitar years ago yet he never pushed it to CRAN.
We
      made changes to Rcpp Modules during this time, we also made changes
to
      how exception escalate (different OS drive this...).  Because this
      package only ever stayed on GitHub this did not get tested.  Else we
      would have known sooner.

 iii) Rcpp Modules is a clever trick that is rather useful. Sadly, it is
not
      that well documented and its principal creator has moved on too so
the
      code is not exactly actively maintained.

So here we need a volunteers to dive it this and debug, maybe bisect with
older Rcpp releases, ...

It looks like this won't be Carlos, and it probably won't be me either.

Dirk

--
http://dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org
_______________________________________________
Rcpp-devel mailing list
Rcpp-devel at lists.r-forge.r-project.org
https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel

Qiang Kou
qkou at umail.iu.edu
School of Informatics and Computing, Indiana University
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.r-forge.r-project.org/pipermail/rcpp-devel/attachments/20151119/17c862dd/attachment.html>
| Hi, Simon,
| 
| Is there any specific reason to use this?
| 
| If you need a binding for libgit2, maybe you can try?https://github.com/
| ropensci/git2r
| 
| At least it is actively maintained.

And on CRAN ...

Dirk
http://dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org
On Nov 19, 2015, at 3:59 PM, Qiang Kou <qkou at umail.iu.edu> wrote:

Hi, Simon,

Is there any specific reason to use this?

This is OT, but, yes, git2r didn't exist when we wrote gitgist so we couldn't use it, obviously. Now, someone would have to re-write gitgist to use git2r - if you want to volunteer, I'd be very grateful :).

Thanks,
Simon
If you need a binding for libgit2, maybe you can try https://github.com/ropensci/git2r

At least it is actively maintained.

Best,

KK

On Thu, Nov 19, 2015 at 3:41 PM, Dirk Eddelbuettel <edd at debian.org> wrote:

Simon,

That may be a little tricky for a few reasons:

 i)   We just released Rcpp 0.12.2. I spend a considerable amount of time
      testing prior to a release. This time we had three full reverse
      dependency checks totalling over 1500 builds given the 500+ revdeps.
      Any such issue would have stopped us.

 ii)  AFAIK Carlos wrote guitar years ago yet he never pushed it to CRAN. We
      made changes to Rcpp Modules during this time, we also made changes to
      how exception escalate (different OS drive this...).  Because this
      package only ever stayed on GitHub this did not get tested.  Else we
      would have known sooner.

 iii) Rcpp Modules is a clever trick that is rather useful. Sadly, it is not
      that well documented and its principal creator has moved on too so the
      code is not exactly actively maintained.

So here we need a volunteers to dive it this and debug, maybe bisect with
older Rcpp releases, ...

It looks like this won't be Carlos, and it probably won't be me either.

Dirk

--
http://dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org
_______________________________________________
Rcpp-devel mailing list
Rcpp-devel at lists.r-forge.r-project.org
https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel

-- 
Qiang Kou
qkou at umail.iu.edu
School of Informatics and Computing, Indiana University

Simon, Carlos, Prateek,

It's an issue between R and guitar. Rcpp is an bystander.  

Upon cloning and running R CMD INSTALL we see it end on

*** installing help indices
** building package indices
** testing if installed package can be loaded
Found more than one class "Rcpp_Reference" in cache; using the first, from namespace 'guitar'
Found more than one class "Rcpp_Reference" in cache; using the first, from namespace 'guitar'
Found more than one class "Rcpp_Repository" in cache; using the first, from namespace 'guitar'
Found more than one class "Rcpp_Repository" in cache; using the first, from namespace 'guitar'
Found more than one class "Rcpp_Index" in cache; using the first, from namespace 'guitar'
Found more than one class "Rcpp_Index" in cache; using the first, from namespace 'guitar'
Found more than one class "Rcpp_OID" in cache; using the first, from namespace 'guitar'
Found more than one class "Rcpp_OID" in cache; using the first, from namespace 'guitar'
Found more than one class "Rcpp_ODBObject" in cache; using the first, from namespace 'guitar'
Found more than one class "Rcpp_ODBObject" in cache; using the first, from namespace 'guitar'
Found more than one class "Rcpp_ODB" in cache; using the first, from namespace 'guitar'
Found more than one class "Rcpp_ODB" in cache; using the first, from namespace 'guitar'
Found more than one class "Rcpp_Commit" in cache; using the first, from namespace 'guitar'
Found more than one class "Rcpp_Commit" in cache; using the first, from namespace 'guitar'
Found more than one class "Rcpp_Tree" in cache; using the first, from namespace 'guitar'
Found more than one class "Rcpp_Tree" in cache; using the first, from namespace 'guitar'
Found more than one class "Rcpp_TreeEntry" in cache; using the first, from namespace 'guitar'
Found more than one class "Rcpp_TreeEntry" in cache; using the first, from namespace 'guitar'
Found more than one class "Rcpp_Blob" in cache; using the first, from namespace 'guitar'
Found more than one class "Rcpp_Blob" in cache; using the first, from namespace 'guitar'
Found more than one class "Rcpp_Tag" in cache; using the first, from namespace 'guitar'
Found more than one class "Rcpp_Tag" in cache; using the first, from namespace 'guitar'
* DONE (guitar)
edd at max:~/git/guitar(master)$ 

We had the same thing in the Rcpp examples and did this:

2015-08-26  Dirk Eddelbuettel  <edd at debian.org>

        * inst/unitTests/testRcppClass/src/rcpp_module.cpp: Renamed Module
        'World' to 'RcppClassWorld' to avoid multiple modules with same name
        (pull request #351, fixed issue #350)

        * inst/unitTests/testRcppClass/man/Rcpp_class_examples.Rd: Ditto
        * inst/unitTests/testRcppClass/R/load.R: Ditto
        * inst/unitTests/testRcppClass/NAMESPACE: Ditto
        * inst/unitTests/runit.Module.client.package.R: Re-enable test

        * inst/unitTests/testRcppModule/src/rcpp_module.cpp: Renamed Module
        'World' to 'RcppModuleWorld' to avoid multiple modules with same name

        * inst/unitTests/cpp/modref.cpp: Rename 'World' to 'ModRefWorld'
        * inst/unitTests/runit.modref.R (test.modRef): Ditto

        * inst/unitTests/cpp/Module.cpp: Rename 'World' to 'ModuleWorld'
        * inst/unitTests/runit.Module.R (test.Module): Ditto

2015-08-25  Kurt Hornik  <Kurt.Hornik at wu.ac.at>

        * R/loadModule.R: For now=TRUE, always set .botched to FALSE which
        corresponds to the case of the methods package being in the search path
        * inst/include/Rcpp/Reference.h: Call call with the internal Namespace

Something was changed (by MMae as I recall) in that period that affects the
name lookup and you simply cannot more than one reference to the same name.
I have not yet looked into guitar -- it may just be that its build is weird
and symbols appear multiple times.

But "someone" needs to disentangle this.

Dirk
http://dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org
| 
| Simon, Carlos, Prateek,
| 
| It's an issue between R and guitar. Rcpp is an bystander.  
| 
| Upon cloning and running R CMD INSTALL we see it end on
| 
| *** installing help indices
| ** building package indices
| ** testing if installed package can be loaded
| Found more than one class "Rcpp_Reference" in cache; using the first, from namespace 'guitar'
| Found more than one class "Rcpp_Reference" in cache; using the first, from namespace 'guitar'
| Found more than one class "Rcpp_Repository" in cache; using the first, from namespace 'guitar'
| Found more than one class "Rcpp_Repository" in cache; using the first, from namespace 'guitar'
| Found more than one class "Rcpp_Index" in cache; using the first, from namespace 'guitar'
| Found more than one class "Rcpp_Index" in cache; using the first, from namespace 'guitar'
| Found more than one class "Rcpp_OID" in cache; using the first, from namespace 'guitar'
| Found more than one class "Rcpp_OID" in cache; using the first, from namespace 'guitar'
| Found more than one class "Rcpp_ODBObject" in cache; using the first, from namespace 'guitar'
| Found more than one class "Rcpp_ODBObject" in cache; using the first, from namespace 'guitar'
| Found more than one class "Rcpp_ODB" in cache; using the first, from namespace 'guitar'
| Found more than one class "Rcpp_ODB" in cache; using the first, from namespace 'guitar'
| Found more than one class "Rcpp_Commit" in cache; using the first, from namespace 'guitar'
| Found more than one class "Rcpp_Commit" in cache; using the first, from namespace 'guitar'
| Found more than one class "Rcpp_Tree" in cache; using the first, from namespace 'guitar'
| Found more than one class "Rcpp_Tree" in cache; using the first, from namespace 'guitar'
| Found more than one class "Rcpp_TreeEntry" in cache; using the first, from namespace 'guitar'
| Found more than one class "Rcpp_TreeEntry" in cache; using the first, from namespace 'guitar'
| Found more than one class "Rcpp_Blob" in cache; using the first, from namespace 'guitar'
| Found more than one class "Rcpp_Blob" in cache; using the first, from namespace 'guitar'
| Found more than one class "Rcpp_Tag" in cache; using the first, from namespace 'guitar'
| Found more than one class "Rcpp_Tag" in cache; using the first, from namespace 'guitar'
| * DONE (guitar)
| edd at max:~/git/guitar(master)$ 
| 
| We had the same thing in the Rcpp examples and did this:
| 
| 2015-08-26  Dirk Eddelbuettel  <edd at debian.org>
| 
|         * inst/unitTests/testRcppClass/src/rcpp_module.cpp: Renamed Module
|         'World' to 'RcppClassWorld' to avoid multiple modules with same name
|         (pull request #351, fixed issue #350)
| 
|         * inst/unitTests/testRcppClass/man/Rcpp_class_examples.Rd: Ditto
|         * inst/unitTests/testRcppClass/R/load.R: Ditto
|         * inst/unitTests/testRcppClass/NAMESPACE: Ditto
|         * inst/unitTests/runit.Module.client.package.R: Re-enable test
| 
|         * inst/unitTests/testRcppModule/src/rcpp_module.cpp: Renamed Module
|         'World' to 'RcppModuleWorld' to avoid multiple modules with same name
| 
|         * inst/unitTests/cpp/modref.cpp: Rename 'World' to 'ModRefWorld'
|         * inst/unitTests/runit.modref.R (test.modRef): Ditto
| 
|         * inst/unitTests/cpp/Module.cpp: Rename 'World' to 'ModuleWorld'
|         * inst/unitTests/runit.Module.R (test.Module): Ditto
| 
| 2015-08-25  Kurt Hornik  <Kurt.Hornik at wu.ac.at>
| 
|         * R/loadModule.R: For now=TRUE, always set .botched to FALSE which
|         corresponds to the case of the methods package being in the search path
|         * inst/include/Rcpp/Reference.h: Call call with the internal Namespace
| 
| 
| Something was changed (by MMae as I recall) in that period that affects the
| name lookup and you simply cannot more than one reference to the same name.
| I have not yet looked into guitar -- it may just be that its build is weird
| and symbols appear multiple times.
| 
| But "someone" needs to disentangle this.

It used some guerilla code that "knew better than we did" for loading; all
that stuff changed way back when in 2012 (?) or so -- during Rcpp 0.9.*.

This was simply stale code.  With a simple change, it now passes R CMD check,
for a suitable definition of 'passes'.

I'll send you a pull request in a minute or two.

Dirk
http://dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org
Many thanks, Dirk! It does indeed fix the problem, very much appreciated!
Simon

On 21 November 2015 at 15:47, Dirk Eddelbuettel wrote:
| 
| Simon, Carlos, Prateek,
| 
| It's an issue between R and guitar. Rcpp is an bystander.  
| 
| Upon cloning and running R CMD INSTALL we see it end on
| 
| *** installing help indices
| ** building package indices
| ** testing if installed package can be loaded
| Found more than one class "Rcpp_Reference" in cache; using the first, from namespace 'guitar'
| Found more than one class "Rcpp_Reference" in cache; using the first, from namespace 'guitar'
| Found more than one class "Rcpp_Repository" in cache; using the first, from namespace 'guitar'
| Found more than one class "Rcpp_Repository" in cache; using the first, from namespace 'guitar'
| Found more than one class "Rcpp_Index" in cache; using the first, from namespace 'guitar'
| Found more than one class "Rcpp_Index" in cache; using the first, from namespace 'guitar'
| Found more than one class "Rcpp_OID" in cache; using the first, from namespace 'guitar'
| Found more than one class "Rcpp_OID" in cache; using the first, from namespace 'guitar'
| Found more than one class "Rcpp_ODBObject" in cache; using the first, from namespace 'guitar'
| Found more than one class "Rcpp_ODBObject" in cache; using the first, from namespace 'guitar'
| Found more than one class "Rcpp_ODB" in cache; using the first, from namespace 'guitar'
| Found more than one class "Rcpp_ODB" in cache; using the first, from namespace 'guitar'
| Found more than one class "Rcpp_Commit" in cache; using the first, from namespace 'guitar'
| Found more than one class "Rcpp_Commit" in cache; using the first, from namespace 'guitar'
| Found more than one class "Rcpp_Tree" in cache; using the first, from namespace 'guitar'
| Found more than one class "Rcpp_Tree" in cache; using the first, from namespace 'guitar'
| Found more than one class "Rcpp_TreeEntry" in cache; using the first, from namespace 'guitar'
| Found more than one class "Rcpp_TreeEntry" in cache; using the first, from namespace 'guitar'
| Found more than one class "Rcpp_Blob" in cache; using the first, from namespace 'guitar'
| Found more than one class "Rcpp_Blob" in cache; using the first, from namespace 'guitar'
| Found more than one class "Rcpp_Tag" in cache; using the first, from namespace 'guitar'
| Found more than one class "Rcpp_Tag" in cache; using the first, from namespace 'guitar'
| * DONE (guitar)
| edd at max:~/git/guitar(master)$ 
| 
| We had the same thing in the Rcpp examples and did this:
| 
| 2015-08-26  Dirk Eddelbuettel  <edd at debian.org>
| 
|         * inst/unitTests/testRcppClass/src/rcpp_module.cpp: Renamed Module
|         'World' to 'RcppClassWorld' to avoid multiple modules with same name
|         (pull request #351, fixed issue #350)
| 
|         * inst/unitTests/testRcppClass/man/Rcpp_class_examples.Rd: Ditto
|         * inst/unitTests/testRcppClass/R/load.R: Ditto
|         * inst/unitTests/testRcppClass/NAMESPACE: Ditto
|         * inst/unitTests/runit.Module.client.package.R: Re-enable test
| 
|         * inst/unitTests/testRcppModule/src/rcpp_module.cpp: Renamed Module
|         'World' to 'RcppModuleWorld' to avoid multiple modules with same name
| 
|         * inst/unitTests/cpp/modref.cpp: Rename 'World' to 'ModRefWorld'
|         * inst/unitTests/runit.modref.R (test.modRef): Ditto
| 
|         * inst/unitTests/cpp/Module.cpp: Rename 'World' to 'ModuleWorld'
|         * inst/unitTests/runit.Module.R (test.Module): Ditto
| 
| 2015-08-25  Kurt Hornik  <Kurt.Hornik at wu.ac.at>
| 
|         * R/loadModule.R: For now=TRUE, always set .botched to FALSE which
|         corresponds to the case of the methods package being in the search path
|         * inst/include/Rcpp/Reference.h: Call call with the internal Namespace
| 
| 
| Something was changed (by MMae as I recall) in that period that affects the
| name lookup and you simply cannot more than one reference to the same name.
| I have not yet looked into guitar -- it may just be that its build is weird
| and symbols appear multiple times.
| 
| But "someone" needs to disentangle this.

It used some guerilla code that "knew better than we did" for loading; all
that stuff changed way back when in 2012 (?) or so -- during Rcpp 0.9.*.

This was simply stale code.  With a simple change, it now passes R CMD check,
for a suitable definition of 'passes'.

I'll send you a pull request in a minute or two.

Dirk

-- 
http://dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org