Prev 1384 / 2152 Next

mpirun and R

Jonathan Greenberg

Sat, Jun 2, 2012 1:55 PM

Sorry, short followup: if I keep the NO_AFFINITY=1 commented, I don't
get the failure below, but I get the aforementioned single-threaded
cluster execution.  I tried this:

# NO_AFFINITY=1 LEFT COMMENTED IN OPENBLAS MAKEFILE.RULE
require(parallel)
mcaffinity()
# It shows, of course, "1"
mcaffinity(1:12)

Loading required package: Rmpi
[mynode:00415] [[47733,1],0] routed:binomial: Connection to lifeline
[[47733,0],0] lost

So it seems like setting the affinity causes Rmpi to choke...

--j

On Sat, Jun 2, 2012 at 3:39 PM, Jonathan Greenberg <jgrn at illinois.edu> wrote:

Ok, moving along here. ?So I got R-devel installed and by switching
back and forth between a fresh install of openblas (with the
NO_AFFINITY = 1 line uncommented) I can get a parallel BLAS, but when
trying an Rmpi call with that BLAS selected I get (under OpenMPI):

require("snow")

Loading required package: snow

cl <- makeCluster(12,type="MPI")

Loading required package: Rmpi
[mynode:10521] [[37603,1],0] routed:binomial: Connection to lifeline
[[37603,0],0] lost

Note that I get the same error if I additionally uncomment
USE_OPENMP=1 when I compile openblas as well.

By setting the libRblas.so that comes with R (I'm using
update-alternatives to swap back and forth for testing), I can run
Rmpi correctly, but I am left with a single threaded BLAS.

Given that I can't install binary packages on the system I'm using
(everything I'm using has to be compiled "by hand" -- this is a system
policy, and nothing I can do to get around it), is there any way to
get a multithreaded BLAS working AND Rmpi behaving properly? ?If I
need to be using a different BLAS, which is recommended? ?ATLAS? ?Is
there a way to get the Intel MKL working with R?

Thanks!

--j


On Sat, Jun 2, 2012 at 1:04 PM, Simon Urbanek
<simon.urbanek at r-project.org> wrote:

On Jun 2, 2012, at 1:30 PM, Jonathan Greenberg wrote:

Simon:

Thanks -- I'm afraid I'm a little unclear how to go about grabbing the updated parallel (since my version running on R 2.15 doesn't have mcaffinity) -- is there a download link to it?

No, the official way is to simply use R-devel. However, you should be able to merge r59189 from R-devel to R-2.15 with no conflicts if you wish.

FWIW the link to the original thread which includes the other solutions is at
https://stat.ethz.ch/pipermail/r-sig-hpc/2012-April/001357.html

Cheers,
Simon

On Sat, Jun 2, 2012 at 12:20 PM, Simon Urbanek
<simon.urbanek at r-project.org> wrote:

On Jun 2, 2012, at 12:46 PM, Jonathan Greenberg wrote:

Steve:

It was built with OpenBLAS,

That is the problem - this was previously discussed here, see the archive. OpenBLAS changes the affinity of the process to use only one CPU. You have to reset the affinity either with Linux tools or using mcaffinity (available in R-devel).

Cheers,
Simon

but does that matter with an MPI-based
function (i.e. I thought GotoBLAS was an entirely different hpc aspect
that is only used for linear algebra routines) -- but yes, all the
spawned R processes end up spawning on a single cpu, but if I use
mpirun it functions properly. ?I had to "roll" OpenBLAS myself on this
system, because it only has Intel MKL installed by the admins which I
have yet to get to play right with R. ?OpenBLAS does work for LA
commands tho.

# In fact, running this on the normal spawned R uses all cores:
a = matrix(rnorm(5000*5000), 5000, 5000)
b = matrix(rnorm(5000*5000), 5000, 5000)
c = a%*%b

#But then in the same instance running:
require(raster)
beginCluster()
# Only spawns on one core.

Are there "better" parameters I might pass to snow to get this
working? ?I get the same behavior in snowfall and sfInit():

require(snowfall)
sfInit(parallel=TRUE,cpus=12)
sfStop()
# All spawns execute on a single CPU
sfInit(parallel=TRUE,cpus=12,type="MPI")
sfStop()
# All spawns execute on a single CPU

Incidentally (and I don't consider this a perfectly satisfactory
answer, so please continue to give me some advice to try out), this
command at least lets me run R in interactive mode and doesn't always
bail when I type in an incorrect statement:

`which mpirun` -n 1 -machinefile $PBS_NODEFILE R --interactive
(note the --interactive instead of the --vanilla)

With that said if I need to kill a process with control-c (usually
just returning me to an R prompt) I do get R bailing back to the bash
command line. ?The other reason I'd like a within-R solution to this
is that I do my development within the Stat-et/Eclipse environment,
and (at least right now) there is no way to modify how it launches R
remotely.

--j

On Fri, Jun 1, 2012 at 3:12 PM, Stephen Weston
<stephen.b.weston at gmail.com> wrote:

So you wanted 12 cpus on a single node, but the 12 spawned
R processes were all scheduled by your OS on a single cpu
rather than multiple cpus/cores on that node?

If so, that suggests that somehow the cpu affinity has been set.
We've seen this type of problem when using GotoBLAS2/OpenBLAS.
Has your R installation been built with either of them?

- Steve


On Fri, Jun 1, 2012 at 11:52 AM, Jonathan Greenberg <jgrn at illinois.edu> wrote:

R-sig-hpc'ers:

Our system (running openmpi) allows for an interactive session to be
created with N number of CPUs allotted to it (12 in my case). ?Here's
the qsub command to get the interactive node running:

qsub -X -I -q [mygroup] -l nodes=1:ppn=12,walltime=48:00:00

If I boot R and then try some HPC R commands e.g.:

require(raster)
# Note this is just a wrapper for a snow call:
beginCluster()

I get:

beginCluster()

Loading required package: snow
12 cores detected
cluster type: MPI
Loading required package: Rmpi
? ? ? ?12 slaves are spawned successfully. 0 failed.

If I "top" I see that I have 12 (13?) R spawns running. ?The problem
is, they are all running on a SINGLE cpu, not distributed amongst all
12 cpus (even though it detected it). ?My first question is: why is
this? ?Is there a way to fix this from a standard "R" launch?

Now, I can SOMEWHAT fix this by:
`which mpirun` -n 1 -machinefile $PBS_NODEFILE R --vanilla

When I run the same commands, they distribute properly to all 12 cpus
BUT ANY error I make in typing will cause the entire system to "die":

require(raster)

require(raster)
Loading required package: raster
Loading required package: sp
raster 1.9-92 (1-May-2012)

beginCluster()

beginCluster()
Loading required package: snow
12 cores detected
cluster type: MPI
Loading required package: Rmpi
? ? ? ?12 slaves are spawned successfully. 0 failed.

abc

Error: object 'abc' not found
Execution halted
--------------------------------------------------------------------------
mpirun has exited due to process rank 0 with PID 28932 on
node [mynode] exiting without calling "finalize". This may
have caused other processes in the application to be
terminated by signals sent by mpirun (as reported here).
--------------------------------------------------------------------------

Is there a way to allow me a "safer" mpirun launch that won't die if I
make a small typo? ?This makes it REALLY hard to troubleshoot code if
any little error causes the quit.

--j


--
Jonathan A. Greenberg, PhD
Assistant Professor
Department of Geography and Geographic Information Science
University of Illinois at Urbana-Champaign
607 South Mathews Avenue, MC 150
Urbana, IL 61801
Phone: 415-763-5476
AIM: jgrn307, MSN: jgrn307 at hotmail.com, Gchat: jgrn307, Skype: jgrn3007
http://www.geog.illinois.edu/people/JonathanGreenberg.html

_______________________________________________
R-sig-hpc mailing list
R-sig-hpc at r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-hpc

_______________________________________________
R-sig-hpc mailing list
R-sig-hpc at r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-hpc

Jonathan A. Greenberg, PhD
Assistant Professor
Department of Geography and Geographic Information Science
University of Illinois at Urbana-Champaign
607 South Mathews Avenue, MC 150
Urbana, IL 61801
Phone: 415-763-5476
AIM: jgrn307, MSN: jgrn307 at hotmail.com, Gchat: jgrn307, Skype: jgrn3007
http://www.geog.illinois.edu/people/JonathanGreenberg.html

Thread (16 messages)

Jonathan Greenberg mpirun and R Jun 1 Stephen Weston mpirun and R Jun 1 Jonathan Greenberg mpirun and R Jun 2 Simon Urbanek mpirun and R Jun 2 Jonathan Greenberg mpirun and R Jun 2 Simon Urbanek mpirun and R Jun 2 Jonathan Greenberg mpirun and R Jun 2 Jonathan Greenberg mpirun and R Jun 2 Jonathan Greenberg mpirun and R Jun 2 Simon Urbanek mpirun and R Jun 2 Simon Urbanek mpirun and R Jun 2 Jonathan Greenberg mpirun and R Jun 2 Simon Urbanek mpirun and R Jun 2 Jonathan Greenberg mpirun and R Jun 2 Simon Urbanek mpirun and R Jun 2 Jonathan Greenberg mpirun and R Jun 3