SNOW Hybrid Cluster in R, Network problems

Wed, Jul 4, 2012 12:23 AM

Hi Steve,

as far as I know rmpi is only available for Unix. So I would say I am a 
Windows enthusiast, may be this is not optimal, but in the end MPI is 
ending with the missing package support in R. Please correct me, if I 
were wrong. And the final question is, if it is possible to build a 
hybrid cluster with MPI and R. For Socket clusters the answer is "yes".


Cu
Martin

Dipl.-Ing. Martin Seilmayer

Helmholtz-Zentrum Dresden-Rossendorf e. V.

Institut fuer Fluiddynamik
Abteilung Magnetohydrodynamik
Bautzner Landstra?e 400
01328 Dresden, Germany

@fon: +49 351 260  3165
@fax: +49 351 260 12969
@web: www.hzdr.de

Am 03.07.2012 20:03, schrieb Stephen Weston:

Hi Martin,

So if mpiexec works on your machines, why aren't you creating
a snow MPI cluster?

- Steve


On Tue, Jul 3, 2012 at 11:09 AM, Martin Seilmayer <M.Seilmayer at hzdr.de> wrote:

Hi Steve,

:) so you pointed out the most interesting problem in this challenge: "How
to start (fast) a remote process on Windows from a console"

The aim was to find a clean solution without mixing up Windows with Linux
staff like ssh or PsTools or rsh or  vis versa. With MPICH2, smpd
service/daemon running, it is possible so start processes on each worker.
I am working on a Windows machine, so the R command to start a
Windows-worker is as follows:
     system(paste("mpiexec -hosts ",length(nodestostart),"
",paste(nodestostart,collapse=" ")," C:\\MPI_R_Start.bat
",hostname,sep=""),wait=F)
"wait = F" important, because the next command should be
makeCluster(listofnodes,type=SOCK)
So I wrote a batch-file for windows and a shell script for Linux, which sets
up each machine individually and which is started remote form MPICH2. This
config script starts as many Rscripts as I had configured before. The Master
on the other side knows how many workers are available an each machine.

Important! It takes no configuration like rshcmd (because ssh is not
available on Win). One must guarantee that "makeCluster()" comes after
"starting Workers" thats the "trick". This works fine, because makeCluster()
makes the master waiting for each worker calling back. If the master is not
responding, the worker shuts down, so far in the Linux world.

And finally: Yes Mpich2 is able to start every program / command on a remote
machine. Thats a bit of a security problem, if you were very strong and
restrictive to me.

I hope I answered in your sense!

Martin



Dipl.-Ing. Martin Seilmayer

Helmholtz-Zentrum Dresden-Rossendorf e. V.

Institut fuer Fluiddynamik
Abteilung Magnetohydrodynamik
Bautzner Landstra?e 400
01328 Dresden, Germany

@fon: +49 351 260 3165
@fax: +49 351 260 12969
@web: www.hzdr.de

Am 03.07.2012 16:44, schrieb Stephen Weston:

On Tue, Jul 3, 2012 at 5:31 AM, Martin Seilmayer <M.Seilmayer at hzdr.de>
wrote:

Hi all of you,

I successfully created a hybrid cluster of several Windows and Linux
machines using snow and MPICH2. Basically I setup a SOCK - Cluster. To
start
the Rscript processes on each machine MPICH2 comes in the game. Because
it

Are you saying that you're using MPICH2 to start your workers on the
remote machines when creating a SOCK cluster?  How are you doing that?
Are you setting the rshcmd option in some way?  Does MPICH2 include a
remote execution command that is ssh-like?

- Steve Weston

SNOW Hybrid Cluster in R, Network problems

Thread (7 messages)