Skip to content

Request for help with UBSAN and total absense of CRAN response

8 messages · Dirk Eddelbuettel, Avraham Adler, Dan Tenenbaum +2 more

Original
Dirk Eddelbuettel
# 
CRAN has a package of mine in upload limbo because it failed UBSAN.

I am not entirely ignorant on the topic of sanitizers and SAN / ASAN / UBSAN;
we created not one but two Docker containers with ASAN and USBAN:

   https://registry.hub.docker.com/u/rocker/r-devel-san/
   https://registry.hub.docker.com/u/rocker/r-devel-ubsan-clang/

as well as predecessors to them in earlier Docker repos.

Yet I fail to recreate the errors reported by CRAN:

   http://www.stats.ox.ac.uk/pub/bdr/memtests/UBSAN-clang-trunk/RcppAnnoy/tests/runUnitTests.Rout
   http://www.stats.ox.ac.uk/pub/bdr/memtests/UBSAN/RcppAnnoy/tests/runUnitTests.Rout

I asked politely (and twice) for help with the corresponding compiler
configuration(s).  But CRAN is of course way above communicating with mere
mortals such as yours truly.

So I have no recourse other than to spam all of you: if anybody here has a
working UBSAN setup which can replicate the issue seen in the (rather small)
RcppAnnoy package?

Erik (upstream for Annoy, CC'ed) and I would be most grateful.  We do not
like being held hostage on an error report we cannot replicate and for which
we do not receive any help (or even further communication) whatsoever.

Dirk
about to turn into yet another frustrated CRAN user
Avraham Adler
# 
Hello, Dirk.

I have no experience with UBSAN etc., but as you mention not being
able to recreate the CRAN errors, it reminds me of the time I had
trouble getting my pacakge approved for CRAN as it passed all tets but
Win32, and I have a Win64 setup and could not test it. Have you tried
submitting the package to
<http://win-builder.r-project.org/upload.aspx> and seeing if the CRAN
errors get created there, or is the problem CRAN shows on platforms
other than Windows?

Avi
On Tue, Jan 13, 2015 at 10:30 AM, Dirk Eddelbuettel <edd at debian.org> wrote:
Dan Tenenbaum
# 
----- Original Message -----
Where should the package source be downloaded from? I see it in CRAN (but presumably the latest version that causes the issue is not yet downloadable) and in github.

Dan
Dirk Eddelbuettel
#
On 13 January 2015 at 08:21, Dan Tenenbaum wrote:
| Where should the package source be downloaded from? I see it in CRAN (but presumably the latest version that causes the issue is not yet downloadable) and in github.

The "presumable" assumption is incorrect AFAIK. 

The error should presumably came up in both versions as annoylib.h did not
change there.  Feel free to prove me wrong :) and just use whatever is easiest.

Dirk
Martyn Plummer
#
On Tue, 2015-01-13 at 10:34 -0600, Dirk Eddelbuettel wrote:
This is a curious case. 

Here is where the first error occurs:

Executing test function test01getNNsByVector  ... 
Breakpoint 1, 0x00000000009c0440 in __ubsan_handle_out_of_bounds ()
(gdb) frame 1
#1  0x00007fffe777935b in AnnoyIndex<int, float, Angular<int, float> >::_get_all_nns (this=0x3a7e8f0, v=0x37d95d0, n=3, result=0x7ffffffee1e8)
    at ../inst/include/annoylib.h:532
532		nns.insert(nns.end(), nd->children, &(nd->children[nd->n_descendants]));
(gdb) p nd->children
$48 = {0, 1}
(gdb) p nd->n_descendants
$49 = 3
(gdb) p nns
$50 = std::vector of length 0, capacity 0

So we are trying to insert 3 values from an array of length 2 into an
STL vector. 

Comments in the header file annoylib.h (lines 114-130) show that this is
a result of a "memory optimization". Small objects have a completely
different format but are allocated in the same memory. When the
optimization is used the array is deliberately allowed to overflow:

    S children[2]; // Will possibly store more than 2
    T v[1]; // We let this one overflow intentionally

A workaround is to turn off the optimization by reducing the threshold
for the alternate data format (_K) to such a low level that it is never
used (annoylib.h, line 259):

    //_K = (sizeof(T) * f + sizeof(S) * 2) / sizeof(S);
    _K = 2; //Turn off memory optimization

I think this is a case of "being too clever by half".

Martyn



-----------------------------------------------------------------------
This message and its attachments are strictly confidenti...{{dropped:8}}
1 day later
Jan van der Laan
# 
Dirk,

The vagrant setup I use to test my packages with UBSAN also seems to  
replicate the error reported by CRAN (together with some other  
warnings). I have attached the files (I hope they get through the  
filters). I suppose you know what to do with them.

Jan





Dirk Eddelbuettel <edd at debian.org> schreef:
1 day later
Jan van der Laan
# 
... and they didn't make it through. I put the files in a gist:

https://gist.github.com/djvanderlaan/1e9beb75d2d595824efc

Jan
On 16-01-15 15:21, Jan van der Laan wrote:
Dirk Eddelbuettel
# 
I would like to express my deepest gratitude to the CRAN Maintainers for
keeping up with their perfect record of never responding to questions from
mere mortals like myself. The ensuing utter and complete silence allowed me
to work with a much higher concentration level.

More seriously, I owe a big thank you to Martyn Plummer and Jan van der Laan
for replying here (and in off-list follow-ups), and to Jeff Horner who
offered help via Twitter and off-list emails.  Thanks to their help, I did
get to the bottom of this, and it seems that part of (either mine only, or a
more general) confusion was that I wanted the test to actually _abort_ on
error.  Which, as I found after too many failed tries, requires also setting
the -fno-sanitize-recover option as well.

Long story short, we now have a working "appliance" to test this via Docker.

This is building on some of the examples I showed at the end of my useR! 2014
presentation, the work Carl and I have been doing in the Rocker.org repo for
Docker containers for R, some experimentation, and an only-at-GitHub-yet new
script for littler.  I hope to blog about this in some more detail, but if
you have Docker set-up, pull the rocker/r-devel-ubsan-clang container.  

Then:

i)  to replicate the reported error:

    docker run --rm -ti -v $(pwd):/mnt rocker/r-devel-ubsan-clang \
           check.r --setwd /mnt --install-deps RcppAnnoy_0.0.5.tar.gz 

    which runs the container with 
      - post-run cleanup (--rm), 
      - terminal and interactive mode (-ti), 
      - the current directory mounted as /mnt in the container (-v ...),
      - invoking the check.r script (from littler as on GitHub, in the path in the
        container) with options to install package dependencies
      - the known-bad tarball

ii) to see a fix not triggering it

    docker run --rm -ti -v $(pwd):/mnt rocker/r-devel-ubsan-clang \
           check.r --setwd /mnt --install-deps RcppAnnoy_0.0.5.1.tar.gz 

    as above but using an updated tarball not showing the error.

This is generic: you can plug any of your R package tarballs in there
and it will run the test.  [ This assumes dependencies can be satisfied via
install.packages(); need to add a hook for apt-get as needed etc pp ]

The UBSAN config I settled on for now is close to what Martyn had sent me (in
off-list follow-up):

 CC="clang-3.5 -fsanitize=undefined \
      -fno-sanitize=float-divide-by-zero,vptr,function -fno-sanitize-recover" 
 CXX="clang++-3.5 -fsanitize=undefined \
      -fno-sanitize=float-divide-by-zero,vptr,function -fno-sanitize-recover"
 CXX1X="clang++-3.5 -fsanitize=undefined \
      -fno-sanitize=float-divide-by-zero,vptr,function -fno-sanitize-recover" 

We may want to turn on other options; Jeff Horner sent me his config which
has a lot more goodies enabled.  

I'd welcome follow-up by anyone interested in working on this, either by
(off-list) email or directly over at the GitHub repo for this container:
   https://github.com/rocker-org/r-devel-san-clang

Thanks,  Dirk