Skip to content

[Bioc-devel] differences between petty and perceval (OS X 10.6.8 build machines for release/devel)

5 messages · Michael Stadler, Kasper Daniel Hansen, Dan Tenenbaum

#
Hi Dan,

I'm cc'ing the list; maybe somebody else has experienced differences
between petty and perceval.

Rbowtie release (1.4.5) is not building under OS X 10.6.8 (petty).

Rbowtie release (1.4.5) and development (1.5.5) are virtually identical
(only DESCRIPTION and NEWS differ).

The development version builds without problems on perceval, but the
release version fails on petty:
http://bioconductor.org/checkResults/devel/bioc-LATEST/Rbowtie/perceval-buildsrc.html
http://bioconductor.org/checkResults/release/bioc-LATEST/Rbowtie/petty-buildsrc.html

The only difference I can make out from the node info pages is that
perceval has an additional section on "C++11 compiler" that is lacking
from petty's NodeInfo page.

Unfortunately, I cannot reproduce the issue, both Rbowtie 1.4.5 and
1.5.5 build successfully under OS X 10.6.8 and 10.7.5 using llvm-gcc-4.2.

Do you have any idea what else could be different between petty and
perceval?

Thank you,
Michael
#
When BioC 2.14 was released, it did work on OS X 10.6.8, but not on 10.9
and neither on some flavours of 32-bit Linux.

The bowtie developers have released version 1.0.1 which addresses some
of these issues. For the remaining one that still made it fail on OS X
10.9, I have been in contact with them and they provided a patch (see
http://sourceforge.net/p/bowtie-bio/bugs/312/) which seems to fix it on
the OS X machines that I have access to, but apparently not for petty.

Going back to the version at release is not an option, since I rate
support for OS X 10.9 more important, especially in the long run.

Michael
On 13.06.2014 15:19, Kasper Daniel Hansen wrote:

  
    
#
Hi Michael,



----- Original Message -----
Martin and Nate and I took a look at this. I managed to come up with a bowtie command line that would reliably reproduce the segfault on petty.

Then we ran that under gdb (and turned off compiler optimizations) and came up with this, which may or may not help you:

petty:vignettes biocbuild$ gdb --args '/Library/Frameworks/R.framework/Versions/3.1/Resources/library/Rbowtie/bowtie' -y -S -k 10 -m 10 -v 2 -r -p 4 --best --strata 'doit/refsIndex/index' 'doit/SpliceMapTemp_876c378e20ac/25mers.map' 'doit/SpliceMapTemp_876c378e20ac/25mers.map_unsorted' 
GNU gdb 6.3.50-20050815 (Apple version gdb-1708) (Mon Aug 15 16:03:10 UTC 2011)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "x86_64-apple-darwin"...Reading symbols for shared libraries ... done

(gdb) run
Starting program: /Library/Frameworks/R.framework/Versions/3.1/Resources/library/Rbowtie/bowtie -y -S -k 10 -m 10 -v 2 -r -p 4 --best --strata doit/refsIndex/index doit/SpliceMapTemp_876c378e20ac/25mers.map doit/SpliceMapTemp_876c378e20ac/25mers.map_unsorted
Reading symbols for shared libraries ++. done

Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_INVALID_ADDRESS at address: 0x23d0d92d
[Switching to process 36144 thread 0x20f]
0x000478b1 in Ebwt<seqan::String<seqan::SimpleType<unsigned char, seqan::_Dna>, seqan::Alloc<void> > >::rowL (this=0xbfffda10, l=@0xa300e14) at ebwt.h:1816
1816        return unpack_2b_from_8b(l.side(this->_ebwt)[l._by], l._bp);
(gdb) l
1811    inline int Ebwt<TStr>::rowL(const SideLocus& l) const {
1812        // Extract and return appropriate bit-pair
1813    #ifdef SIXTY4_FORMAT
1814        return (((uint64_t*)l.side(this->_ebwt))[l._by >> 3] >> ((((l._by & 7) << 2) + l._bp) << 1)) & 3;
1815    #else
1816        return unpack_2b_from_8b(l.side(this->_ebwt)[l._by], l._bp);
1817    #endif
1818    }
1819    
1820    /**
(gdb) p this ->_ebwt
$1 = (uint8_t *) 0x4804a00 "\b<2"
(gdb) p *this ->_ebwt
$2 = 8 '\b'
(gdb) p l._by
$3 = 45
(gdb) p l.side     
$4 = &SideLocus::side(unsigned char const*) const
(gdb) p l.side(this->_ebwt)
$5 = (uint8_t *) 0x23d0d900 <Address 0x23d0d900 out of bounds>
(gdb) p l.side(this->_ebwt)[l._by]
Cannot access memory at address 0x23d0d92d
(gdb) p this ->_ebwt
$6 = (uint8_t *) 0x4804a00 "\b<2"
(gdb) 

Running the same command line under valgrind (on a linux box) produces the following:

==14916== Memcheck, a memory error detector
==14916== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al.
==14916== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info
==14916== Command: /home/biocbuild/bbs-2.14-bioc/R/library/Rbowtie/bowtie -y -S -k 10 -m 10 -v 2 -r -p 4 --best --strata doit/refsIndex/index doit/SpliceMapTemp_876c378e20ac/25mers.map doit/SpliceMapTemp_876c378e20ac/25mers.map_unsorted
==14916== 
# reads processed: 24
# reads with at least one reported alignment: 18 (75.00%)
# reads that failed to align: 6 (25.00%)
Reported 18 alignments to 1 output stream(s)
==14916== 
==14916== HEAP SUMMARY:
==14916==     in use at exit: 804 bytes in 7 blocks
==14916==   total heap usage: 647 allocs, 640 frees, 278,139,442 bytes allocated
==14916== 
==14916== 4 bytes in 1 blocks are definitely lost in loss record 1 of 3
==14916==    at 0x52FB1C7: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14916==    by 0x4315DB: HitSink::HitSink(OutFileBuf*, std::string const&, std::string const&, std::string const&, bool, bool, RecalTable*, std::vector<std::string, std::allocator<std::string> >*) (hit.h:332)
==14916==    by 0x414742: _Z6driverIN5seqan6StringINS0_10SimpleTypeIhNS0_4_DnaEEENS0_5AllocIvEEEEEvPKcRKSsSB_RKSt6vectorISsSaISsEESG_SB_.isra.1161.constprop.1242 (sam.h:53)
==14916==    by 0x41719A: bowtie (ebwt_search.cpp:2864)
==14916==    by 0x4060B9: main (bowtie_main.cpp:60)
==14916== 
==14916== 224 bytes in 4 blocks are definitely lost in loss record 2 of 3
==14916==    at 0x52FB1C7: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14916==    by 0x40C4D4: void twoOrThreeMismatchSearchFull<seqan::String<seqan::SimpleType<unsigned char, seqan::_Dna>, seqan::Alloc<void> > >(PairedPatternSource&, HitSink&, Ebwt<seqan::String<seqan::SimpleType<unsigned char, seqan::_Dna>, seqan::Alloc<void> > >&, Ebwt<seqan::String<seqan::SimpleType<unsigned char, seqan::_Dna>, seqan::Alloc<void> > >&, std::vector<seqan::String<seqan::SimpleType<unsigned char, seqan::_Dna5>, seqan::Alloc<void> >, std::allocator<seqan::String<seqan::SimpleType<unsigned char, seqan::_Dna5>, seqan::Alloc<void> > > >&, bool) (ebwt_search.cpp:1873)
==14916==    by 0x413A20: _Z6driverIN5seqan6StringINS0_10SimpleTypeIhNS0_4_DnaEEENS0_5AllocIvEEEEEvPKcRKSsSB_RKSt6vectorISsSaISsEESG_SB_.isra.1161.constprop.1242 (ebwt_search.cpp:2694)
==14916==    by 0x41719A: bowtie (ebwt_search.cpp:2864)
==14916==    by 0x4060B9: main (bowtie_main.cpp:60)
==14916== 
==14916== 576 bytes in 2 blocks are possibly lost in loss record 3 of 3
==14916==    at 0x52F9DB4: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14916==    by 0x46E2074: _dl_allocate_tls (dl-tls.c:297)
==14916==    by 0x550AABC: pthread_create@@GLIBC_2.2.5 (allocatestack.c:571)
==14916==    by 0x46D358: tthread::thread::thread(void (*)(void*), void*) (tinythread.cpp:207)
==14916==    by 0x40C4E7: void twoOrThreeMismatchSearchFull<seqan::String<seqan::SimpleType<unsigned char, seqan::_Dna>, seqan::Alloc<void> > >(PairedPatternSource&, HitSink&, Ebwt<seqan::String<seqan::SimpleType<unsigned char, seqan::_Dna>, seqan::Alloc<void> > >&, Ebwt<seqan::String<seqan::SimpleType<unsigned char, seqan::_Dna>, seqan::Alloc<void> > >&, std::vector<seqan::String<seqan::SimpleType<unsigned char, seqan::_Dna5>, seqan::Alloc<void> >, std::allocator<seqan::String<seqan::SimpleType<unsigned char, seqan::_Dna5>, seqan::Alloc<void> > > >&, bool) (ebwt_search.cpp:1873)
==14916==    by 0x413A20: _Z6driverIN5seqan6StringINS0_10SimpleTypeIhNS0_4_DnaEEENS0_5AllocIvEEEEEvPKcRKSsSB_RKSt6vectorISsSaISsEESG_SB_.isra.1161.constprop.1242 (ebwt_search.cpp:2694)
==14916==    by 0x41719A: bowtie (ebwt_search.cpp:2864)
==14916==    by 0x4060B9: main (bowtie_main.cpp:60)
==14916== 
==14916== LEAK SUMMARY:
==14916==    definitely lost: 228 bytes in 5 blocks
==14916==    indirectly lost: 0 bytes in 0 blocks
==14916==      possibly lost: 576 bytes in 2 blocks
==14916==    still reachable: 0 bytes in 0 blocks
==14916==         suppressed: 0 bytes in 0 blocks
==14916== 
==14916== For counts of detected and suppressed errors, rerun with: -v
==14916== ERROR SUMMARY: 3 errors from 3 contexts (suppressed: 2 from 2)

So maybe those are places to look? Apparently these sorts of memory errors can occur on one machine even when it does not occur on an identically configured machine.

Dan
2 days later
#
Dear Dan, Martin and Nate,

Thank you for looking into it. I guess that is pointing to a problem
within bowtie.

It looks like the EXC_BAD_ACCESS you see on petty in "ebwt.h" is not
reproducible on the other Mac or Linux machines we tried. Is it possible
to run valgrind on petty? That may confirm/rule out if the memory
(de-)allocation issues reported on Linux are related.

I would like to submit a bug-report to the bowtie developers, but am
reluctant to do that without being able to reproduce the problem or test
potential fixed. I would have the options to go through Rbowtie build
cycles, but would have to rely on the assumption that petty will keep
hitting this hickup even with modified bowtie code. The minor
differences between bowtie 1.0.1 and bowtie 1.0.1-bug-312 argue against
that.

I am tempted to stay with the current situtation:
  - OS X before 10.9 needs to use Rbowtie <= 1.4.4
    (based on bowtie 1.0.1)
  - OS X 10.9 onwards and everything else uses Rbowtie >= 1.4.5
    (based on bowtie 1.0.1 /patched bugs-312).

Thanks again for your efforts,
Michael
On 14.06.2014 01:31, Dan Tenenbaum wrote: