Skip to content

Rmpi segfault after install on Ubuntu Hardy Heron

10 messages · Sklyar, Oleg (London), Paul Hewson, Dirk Eddelbuettel +3 more

Original
Mark Kimpel
# 
I just installed Rmpi on my 64-bit Ubuntu Hardy Heron OS and using the
following without errors:
"R CMD INSTALL Rmpi_0.5-5.tar.gz --configure-args=--with-mpi=/usr/lib64/openmpi"

Immediately at library(Rmpi) I get the segfault displayed in my
complete output below. My first thought is that perhaps I used the
wrong library for openmpi, but with my 64 bit install it seemed like a
logical choice and the install went without a hitch.

Two other general comments:
1. Am I addressing this to the correct list or should I use R-help? As
I read the posting guide, I'm not sure.
2. Has anyone considered an R-SIG-HPC list? Anyone besides me interested?

Mark


R version 2.7.0 Patched (2008-05-18 r45728)
Copyright (C) 2008 The R Foundation for Statistical Computing
ISBN 3-900051-07-0

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

Loading required package: datasets
Loading required package: utils
Loading required package: grDevices
Loading required package: graphics
Loading required package: stats
Loading required package: graph
[1] "packages do not need updated"
*** caught segfault ***
address 0x2104008, cause 'memory not mapped'

Traceback:
 1: .Call("mpi_initialize", PACKAGE = "Rmpi")
 2: f(libname, pkgname)
 3: firstlib(which.lib.loc, package)
 4: doTryCatch(return(expr), name, parentenv, handler)
 5: tryCatchOne(expr, names, parentenv, handlers[[1]])
 6: tryCatchList(expr, classes, parentenv, handlers)
 7: tryCatch(expr, error = function(e) {    call <- conditionCall(e)
 if (!is.null(call)) {        if (identical(call[[1]],
quote(doTryCatch)))             call <- sys.call(-4)        dcall <-
deparse(call)[1]        prefix <- paste("Error in", dcall, ": ")
 LONG <- 75        msg <- conditionMessage(e)        sm <-
strsplit(msg, "\n")[[1]]        if (14 + nchar(dcall, type = "w") +
nchar(sm[1], type = "w") >             LONG)             prefix <-
paste(prefix, "\n  ", sep = "")    }    else prefix <- "Error : "
msg <- paste(prefix, conditionMessage(e), "\n", sep = "")
.Internal(seterrmessage(msg[1]))    if (!silent &&
identical(getOption("show.error.messages"),         TRUE)) {
cat(msg, file = stderr())        .Internal(printDeferredWarnings())
}    invisible(structure(msg, class = "try-error"))})
 8: try(firstlib(which.lib.loc, package))
 9: library("Rmpi")

Possible actions:
1: abort (with core dump, if enabled)
2: normal R exit
3: exit R without saving workspace
4: exit R saving workspace
Selection:
Sklyar, Oleg (London)
# 
Does Rmpi work with OpenMPI at all? It calls lamboot on load from LAM
MPI. I would think that mpi_initialize calls a shared library function
from LAM MPI which simply does not exist in OpenMPI or which does
something different from what it expects. If you ever make Rmpi working
with OpenMPI I would be thankful for the solution. It does compile
against OpenMPI though because the header seems to be matching.

Dr Oleg Sklyar
Technology Group
Man Investments Ltd
+44 (0)20 7144 3803
osklyar at maninvestments.com
**********************************************************************
The contents of this email are for the named addressee(s...{{dropped:22}}
Dirk Eddelbuettel
#
On 11 June 2008 at 10:38, Sklyar, Oleg \(MI London\) wrote:
| Does Rmpi work with OpenMPI at all? It calls lamboot on load from LAM

Yes. 

| MPI. I would think that mpi_initialize calls a shared library function
| from LAM MPI which simply does not exist in OpenMPI or which does
| something different from what it expects. If you ever make Rmpi working
| with OpenMPI I would be thankful for the solution. It does compile
| against OpenMPI though because the header seems to be matching.

Debian switched in October 2007 and with Rmpi 0.5-4. Rmpi 0.5-5 definitely
supports it, Hao Yu made some upstream changes.

I have been using it on Ubuntu/x86 at work all that time (but then I was also
testing the newer Open MPI code). You may want Open MPI 1.2.* though it may
work with 1.1.* as well.

Dirk
Paul Hewson
# 
Hello,
I can confirm that Rmpi 0.5.5 works fine with openmpi 1.1

But if (a) the cluster you are running on is set up to allow spawning and (b) you have lam-mpi installed, there is a function within zzz.R which will spawn a lam-mpi process as well which you might need to prevent.

The package README tells you what to do to prevent lam-mpi starting up.   You either hack at the zzz.R, or you set up a file called lamnodes (I think) as a shell script which contains echo lamnodes NULL (or similar) and put this somewhere on your path.   That tells Rmpi not to fire up lammpi, you can run it under mpirun -np 1 -hostfile myhosts R --vanilla --slave CMD BATCH diddlededoo.R &

Best

Paul

-=-=-==-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Paul Hewson
Lecturer in Statistics
University of Plymouth
Drake Circus
Plymouth PL4 8AA

tel ++44(0)1752 232778
email paul.hewson at plymouth.ac.uk
web http://www.plymouth.ac.uk/staff/phewson
Dirk Eddelbuettel
#
On 11 June 2008 at 00:46, Mark Kimpel wrote:
| I just installed Rmpi on my 64-bit Ubuntu Hardy Heron OS and using the
| following without errors:
| "R CMD INSTALL Rmpi_0.5-5.tar.gz --configure-args=--with-mpi=/usr/lib64/openmpi"

And it concluded without warnings or errors? Configure found all the right files?

As Paul suggested, make sure you're not getting it mixed with
exisiting LAM headers or MPICH2 headers or, say, that you're getting your gcc
and icc experiemenst mixed up or ...  ?

I control this by locally re-building the Debian packages for Open MPI. That
way I only keep one -dev package for MPI and can ensure that I do not get
mixups with other MPI installation. But then I do buy into the Package
Management mantra, but not everybody does...
 
| Immediately at library(Rmpi) I get the segfault displayed in my
| complete output below. My first thought is that perhaps I used the
| wrong library for openmpi, but with my 64 bit install it seemed like a
| logical choice and the install went without a hitch.

Check ldd on Rmpi.so to ensure that you linked against what you thought you
should link against.

And if everything else fails, debug the compiled code and run R under the
debuggger.  See the 'R Extensions' manual.
 
| Two other general comments:
| 1. Am I addressing this to the correct list or should I use R-help? As
| I read the posting guide, I'm not sure.
| 2. Has anyone considered an R-SIG-HPC list? Anyone besides me interested?

I'd join.

Dirk
1 day later
Mark Kimpel
# 
Dirk,

Configure concluded without errors. I am not running this on the HPC
cluster with icc, rather a separate local box used for testing and
learning. Its just an 64-bit Ubuntu 8.04 install on a 4-core machine.
LAM is not installed (I checked to make sure).

So, that leaves me with you last two options. Before I make my first
foray into using the R-debugger, I'd like to try you suggestion to
"Check ldd on Rmpi.so" but don't have a clue how to do so. Could you
instruct me?

Thanks to all for the helpful comments.
Mark
On Wed, Jun 11, 2008 at 8:22 AM, Dirk Eddelbuettel <edd at debian.org> wrote:

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Oleg Sklyar
    

    

      
      #
    

  


  

      
simply cd to $R_HOME/library/Rmpi/libs and do on the command line

ldd -r Rmpi.so

this will display you a list of dependencies of Rmpi.so and you can see 
if you find there the libraries that you expected to link against, or 
there will be some libraries from unexpected paths or missing ones

        
Mark Kimpel wrote:

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Martin Morgan
    

    

      
      #
    

  


  

      
Oleg Sklyar wrote:

            

      
      
      
    

        
I think you'll want to

R CMD ldd $R_HOME/library/Rmpi/libs/Rmpi.so

which picks up the R configuration environment, e.g., compare BLAS below:

mtmorgan at lamprey:~/> R CMD ldd $R_HOME/library/stats/libs/stats.so
         libRblas.so => 
/home/mtmorgan/arch/x86_64/R-devel/lib/libRblas.so (0x00002b543f98b000)
         libgfortran.so.2 => /usr/lib64/libgfortran.so.2 
(0x00002b543fc09000)
         libm.so.6 => /lib64/libm.so.6 (0x00002b543fec4000)
         libc.so.6 => /lib64/libc.so.6 (0x00002b5440118000)
         /lib64/ld-linux-x86-64.so.2 (0x0000555555554000)
mtmorgan at lamprey:~/s> ldd $R_HOME/library/stats/libs/stats.so
         libRblas.so => not found
         libgfortran.so.2 => /usr/lib64/libgfortran.so.2 
(0x00002aeabd923000)
         libm.so.6 => /lib64/libm.so.6 (0x00002aeabdbde000)
         libc.so.6 => /lib64/libc.so.6 (0x00002aeabde32000)
         /lib64/ld-linux-x86-64.so.2 (0x0000555555554000)

Martin

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Dirk Eddelbuettel
    

    

      
      #
    

  


  

      
On 12 June 2008 at 18:38, Mark Kimpel wrote:

        
| Dirk,
| 
| Configure concluded without errors. I am not running this on the HPC
| cluster with icc, rather a separate local box used for testing and
| learning. Its just an 64-bit Ubuntu 8.04 install on a 4-core machine.
| LAM is not installed (I checked to make sure).
| 
| So, that leaves me with you last two options. Before I make my first
| foray into using the R-debugger, I'd like to try you suggestion to
| "Check ldd on Rmpi.so" but don't have a clue how to do so. Could you
| instruct me?

ldd is a command so try 'man ldd'; or just use what Oleg and Martin showed
you.

You still told us where you got R, Rmpi, and Open MPI from.  You could just
do

	$ sudo apt-get install r-cran-rmpi

and get Rmpi installed along with its dependencies (eg R and Open MPI).  That
just works on Debian.  It also worked, as I recall, on Ubuntu 'Gutsy' aka 7.10.

However, on Ubuntu 'Hardy' aka 8.04, this may be broken as I discovered
during some testing for the HPC presentation I'll be giving at Use R. I
filed an Ubuntu bug report at

     https://bugs.launchpad.net/ubuntu/+source/binutils/+bug/234837

but no follow-up has happened.

That said, this still all works for me at 

     a) at work which is x86 and all Ubuntu machines

     b) at home where it is x86 and a mix of Debian and Ubuntu machines

_but_ only after I rebuilt openmpi_1.2.5 on an older Gutsy machine as
detailed in the bugreport. Something on Hardy breaks it. 

I am not an Ubuntu maintainer, so I am not sure who to talk to.  I'll forward
a copy of this to the Ubuntu folks.

As an aside, the Debian packages all work.

Dirk
 
| Thanks to all for the helpful comments.
| Mark
|

        
| On Wed, Jun 11, 2008 at 8:22 AM, Dirk Eddelbuettel <edd at debian.org> wrote:

        
| >

        
| > On 11 June 2008 at 00:46, Mark Kimpel wrote:

        
| > | I just installed Rmpi on my 64-bit Ubuntu Hardy Heron OS and using the
| > | following without errors:
| > | "R CMD INSTALL Rmpi_0.5-5.tar.gz --configure-args=--with-mpi=/usr/lib64/openmpi"
| >
| > And it concluded without warnings or errors? Configure found all the right files?
| >
| > As Paul suggested, make sure you're not getting it mixed with
| > exisiting LAM headers or MPICH2 headers or, say, that you're getting your gcc
| > and icc experiemenst mixed up or ...  ?
| >
| > I control this by locally re-building the Debian packages for Open MPI. That
| > way I only keep one -dev package for MPI and can ensure that I do not get
| > mixups with other MPI installation. But then I do buy into the Package
| > Management mantra, but not everybody does...
| >
| > | Immediately at library(Rmpi) I get the segfault displayed in my
| > | complete output below. My first thought is that perhaps I used the
| > | wrong library for openmpi, but with my 64 bit install it seemed like a
| > | logical choice and the install went without a hitch.
| >
| > Check ldd on Rmpi.so to ensure that you linked against what you thought you
| > should link against.
| >
| > And if everything else fails, debug the compiled code and run R under the
| > debuggger.  See the 'R Extensions' manual.
| >
| > | Two other general comments:
| > | 1. Am I addressing this to the correct list or should I use R-help? As
| > | I read the posting guide, I'm not sure.
| > | 2. Has anyone considered an R-SIG-HPC list? Anyone besides me interested?
| >
| > I'd join.
| >
| > Dirk
| >
| > --
| > Three out of two people have difficulties with fractions.
| >
| > ______________________________________________
| > R-devel at r-project.org mailing list
| > https://stat.ethz.ch/mailman/listinfo/r-devel
| >
| 
| 
| 
| -- 
| Mark W. Kimpel MD ** Neuroinformatics ** Dept. of Psychiatry
| Indiana University School of Medicine
| 
| 15032 Hunter Court, Westfield, IN 46074
| 
| (317) 490-5129 Work, & Mobile & VoiceMail
| (317) 663-0513 Home (no voice mail please)
| 
| ______________________________________________
| R-devel at r-project.org mailing list
| https://stat.ethz.ch/mailman/listinfo/r-devel

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Mark Kimpel
    

    

      
      #
    

  


  

      
Dirk,

Did R CMD ldd $R_HOME/library/Rmpi/libs/Rmpi.so as suggested and
things looked okay. Not really wanting to mess with the debugger I
decided to use your prepackaged version. I completely uninstalled all
openmpi packages on my system using Synaptic and deleted my compiled
from source Rmpi from my site-library. Then I executed "sudo apt-get
install r-cran-rmpi" and things seemed to go smoothly. I put a soft
link from the library aptitude installed Rmpi into to my
~/R_HOME/site-library. I get exactly the same error as before. Does
the minimal debugging info below "options(error = recover) provide any
insight?  Mark

            

      
      
      
    

        
Loading required package: Rmpi

 *** caught segfault ***
address 0x2224008, cause 'memory not mapped'

Traceback:
 1: .Call("mpi_initialize", PACKAGE = "Rmpi")
 2: f(libname, pkgname)
 3: firstlib(which.lib.loc, package)
 4: doTryCatch(return(expr), name, parentenv, handler)
 5: tryCatchOne(expr, names, parentenv, handlers[[1]])
 6: tryCatchList(expr, classes, parentenv, handlers)
 7: tryCatch(expr, error = function(e) {    call <- conditionCall(e)
 if (!is.null(call)) {        if (identical(call[[1]],
quote(doTryCatch)))             call <- sys.call(-4)        dcall <-
deparse(call)[1]        prefix <- paste("Error in", dcall, ": ")
 LONG <- 75        msg <- conditionMessage(e)        sm <-
strsplit(msg, "\n")[[1]]        if (14 + nchar(dcall, type = "w") +
nchar(sm[1], type = "w") >             LONG)             prefix <-
paste(prefix, "\n  ", sep = "")    }    else prefix <- "Error : "
msg <- paste(prefix, conditionMessage(e), "\n", sep = "")
.Internal(seterrmessage(msg[1]))    if (!silent &&
identical(getOption("show.error.messages"),         TRUE)) {
cat(msg, file = stderr())        .Internal(printDeferredWarnings())
}    invisible(structure(msg, class = "try-error"))})
 8: try(firstlib(which.lib.loc, package))
 9: library(package, lib.loc = lib.loc, character.only = TRUE,
logical.return = TRUE,     warn.conflicts = warn.conflicts,
keep.source = keep.source,     version = version)
10: require(Rmpi)

Possible actions:
1: abort (with core dump, if enabled)
2: normal R exit
3: exit R without saving workspace
4: exit R saving workspace
Selection: 2
Save workspace image? [y/n/c]: n
mkimpel at mkimpel-xps600:~/R_HOME/site-library-2.7.0/Rmpi$ R

R version 2.7.0 Patched (2008-05-18 r45728)
________________________________
#stuff deleted for brevity
______________________________________

            

      
      
      
    

        
R version 2.7.0 Patched (2008-05-18 r45728)
x86_64-unknown-linux-gnu

locale:
LC_CTYPE=en_US.UTF-8;LC_NUMERIC=C;LC_TIME=en_US.UTF-8;LC_COLLATE=en_US.UTF-8;LC_MONETARY=C;LC_MESSAGES=en_US.UTF-8;LC_PAPER=en_US.UTF-8;LC_NAME=C;LC_ADDRESS=C;LC_TELEPHONE=C;LC_MEASUREMENT=en_US.UTF-8;LC_IDENTIFICATION=C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
[1] graph_1.18.1

loaded via a namespace (and not attached):
[1] cluster_1.11.10 tools_2.7.0



Mark W. Kimpel MD ** Neuroinformatics ** Dept. of Psychiatry
Indiana University School of Medicine

15032 Hunter Court, Westfield, IN 46074

(317) 490-5129 Work, & Mobile & VoiceMail
(317) 663-0513 Home (no voice mail please)