help debugging segfaults - R-devel

Thu, Jun 13, 2002 5:38 AM #

Hi all,

Thanks to Prof. Ripley, Prof. Gentleman, and Simon Wood (did I miss
anyone?).  The problem seemed to have gone away.

Everyone suggested using some malloc debugger (such as Electric Fence).  All
I did was following half of what BDR suggested below, i.e., changing all the
S_alloc() calls to Calloc() and Free().  I didn't get to try efence, and the
problem seems to have disappeared!

As I read it from the R-exts manual, the difference between S_alloc and
Calloc is that S_alloc takes memory from the heap, whereas Calloc takes
memory addtional to the heap.  So can anyone explain what's going on?  It's
still kind of mysterious to me...

Again, thanks very, very much for the help!

(Some day I might try to use .Call rather than .C, but I think I need to get
a bit more familiarity with .Call on simpler things before I try it with
randomForest.)

Regards,
Andy

-----Original Message-----
From: Prof Brian D Ripley [mailto:ripley@stats.ox.ac.uk]
Sent: Wednesday, June 12, 2002 10:02 AM
To: Liaw, Andy
Cc: 'r-devel@stat.math.ethz.ch'
Subject: Re: [R] help debugging segfaults

(Confined to R-devel).

This almost always means that R's memory system (or malloc's) has been
corrupted by array overruns.

Sometimes gctorture(TRUE) helps.  However in your case it's 
more likely
those S_alloc calls, so try (temporarily) replacing them by 
calls to Calloc
and then use something like Purify or `Electric Fence'. to test for
overruns.

On Wed, 12 Jun 2002, Liaw, Andy wrote:

(Sorry for the cross-post--- I wasn't sure which list is more
appropriate...)

Only a few people read R-devel and not R-help.

Hi everyone,

I've run into segfaults when using my randomForest package

on large dataset

(e.g., 100 x 15200) and large number of trees (e.g., ntree=7000 and
mtry=3000).  I'm wondering if anyone can give me some hints

on where to look

for the problem.

The randomForest package mainly consists of two things:

rf.c contains rf(),

a C wrapper function that calls the Fortran subroutines in

rfsub.f that do

most of the work (slightly altered from Breiman's original

code).  All

memory allocations are done in rf.c, using S_alloc().  When

I run random

forest with the data and setting as mentioned above, it was

able to finish

growing the 7000 trees, but segfault when returning from

rf() to R.  GDB

gave the following (gdb prompts removed):

This is just saying it can't allocate the copies for the returned
values of the .C arguments.  I think you might want to consider .Call
given that you are probably using quite large structures.

do_dotCode (call=0x873aff4, op=0x8a5f620, args=0x8a5d010,

env=0x86fd0a4)

    at dotcode.c:1413
1413            break;
1845        PROTECT(ans = allocVector(VECSXP, nargs));
1846        havenames = 0;
1847        if (dup) {
1849            info.cargs = cargs;
1850            info.allArgs = args;
1851            info.nargs = nargs;
1852            info.functionName = buf;
1853            nargs = 0;
1854            for (pargs = args ; pargs != R_NilValue ; pargs =
CDR(pargs)) {
1855                if(argConverters[nargs]) {
1864                    PROTECT(s =

CPtrToRObj(cargs[nargs], CAR(pargs),

which));

Program received signal SIGSEGV, Segmentation fault.
0x080ddc6a in RunGenCollect (size_needed=1515400) at memory.c:1133
1133                    SEXP next = NEXT_NODE(s);

This is obtained on Linux (Mandrake 8.2 w/enterprise kernel

2.4.8) running

on dual P3-866 Xeon with 2GB RAM, using R-1.5.0 compiled

from source.

Any help/hints/comments are greatly appreciated!

Regards,
Andy

Andy I. Liaw, PhD
Biometrics Research          Phone: (732) 594-0820
Merck & Co., Inc.              Fax: (732) 594-1565
P.O. Box 2000, RY70-38            Rahway, NJ 07065
mailto:andy_liaw@merck.com

--------------------------------------------------------------
----------------

Notice: This e-mail message, together with any attachments,

contains information of Merck & Co., Inc. (Whitehouse 
Station, New Jersey, USA) that may be confidential, 
proprietary copyrighted and/or legally privileged, and is 
intended solely for the use of the individual or entity named 
on this message. If you are not the intended recipient, and 
have received this message in error, please immediately 
return this by e-mail and then delete it.

==============================================================
================

-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.
-.-.-.-.-.-.-.-.-

r-help mailing list -- Read

http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html

Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To:

r-help-request@stat.math.ethz.ch

_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._.
_._._._._._._._._

-- 
Brian D. Ripley,                  ripley@stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272860 (secr)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595

------------------------------------------------------------------------------
Notice: This e-mail message, together with any attachments, contains information of Merck & Co., Inc. (Whitehouse Station, New Jersey, USA) that may be confidential, proprietary copyrighted and/or legally privileged, and is intended solely for the use of the individual or entity named on this message. If you are not the intended recipient, and have received this message in error, please immediately return this by e-mail and then delete it.

==============================================================================

-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
r-devel mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To: r-devel-request@stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._

Duncan Murdoch

Thu, Jun 13, 2002 6:28 AM #

On Thu, 13 Jun 2002 08:38:32 -0400, you wrote in message
<51F9C42DA15CD311BD220008C707D81906FFC35C@usrymx10.merck.com>:

In my experience, that's not such good news.  The bug is probably
still there, it just doesn't have obvious symptoms in the tests you're
using.  I'd still recommend using the malloc debugger to try to track
it down.

Duncan Murdoch
-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
r-devel mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To: r-devel-request@stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._

Thomas Lumley

Thu, Jun 13, 2002 9:05 AM #

On Thu, 13 Jun 2002, Liaw, Andy wrote:

No, the *symptoms* have disappeared. This is not necessarily an
improvement.

It looked as if your code was writing to/reading from memory that didn't
belong to it. What happens then depends on who the memory does belong to,
and what they are doing with it. There's no guarantee that just because it
doesn't crash today that it won't in the future, or worse, start giving
subtly wrong answers.

A segmentation fault is a good thing in this context -- it's a sign that
the operating system has protected you from the consequences of your
actions by killing the program.

	-thomas

-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
r-devel mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To: r-devel-request@stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._