SNOW Hybrid Cluster in R, Network problems
Hi Martin, So if mpiexec works on your machines, why aren't you creating a snow MPI cluster? - Steve
On Tue, Jul 3, 2012 at 11:09 AM, Martin Seilmayer <M.Seilmayer at hzdr.de> wrote:
Hi Steve,
:) so you pointed out the most interesting problem in this challenge: "How
to start (fast) a remote process on Windows from a console"
The aim was to find a clean solution without mixing up Windows with Linux
staff like ssh or PsTools or rsh or vis versa. With MPICH2, smpd
service/daemon running, it is possible so start processes on each worker.
I am working on a Windows machine, so the R command to start a
Windows-worker is as follows:
system(paste("mpiexec -hosts ",length(nodestostart),"
",paste(nodestostart,collapse=" ")," C:\\MPI_R_Start.bat
",hostname,sep=""),wait=F)
"wait = F" important, because the next command should be
makeCluster(listofnodes,type=SOCK)
So I wrote a batch-file for windows and a shell script for Linux, which sets
up each machine individually and which is started remote form MPICH2. This
config script starts as many Rscripts as I had configured before. The Master
on the other side knows how many workers are available an each machine.
Important! It takes no configuration like rshcmd (because ssh is not
available on Win). One must guarantee that "makeCluster()" comes after
"starting Workers" thats the "trick". This works fine, because makeCluster()
makes the master waiting for each worker calling back. If the master is not
responding, the worker shuts down, so far in the Linux world.
And finally: Yes Mpich2 is able to start every program / command on a remote
machine. Thats a bit of a security problem, if you were very strong and
restrictive to me.
I hope I answered in your sense!
Martin
Dipl.-Ing. Martin Seilmayer
Helmholtz-Zentrum Dresden-Rossendorf e. V.
Institut fuer Fluiddynamik
Abteilung Magnetohydrodynamik
Bautzner Landstra?e 400
01328 Dresden, Germany
@fon: +49 351 260 3165
@fax: +49 351 260 12969
@web: www.hzdr.de
Am 03.07.2012 16:44, schrieb Stephen Weston:
On Tue, Jul 3, 2012 at 5:31 AM, Martin Seilmayer <M.Seilmayer at hzdr.de> wrote:
Hi all of you, I successfully created a hybrid cluster of several Windows and Linux machines using snow and MPICH2. Basically I setup a SOCK - Cluster. To start the Rscript processes on each machine MPICH2 comes in the game. Because it
Are you saying that you're using MPICH2 to start your workers on the remote machines when creating a SOCK cluster? How are you doing that? Are you setting the rshcmd option in some way? Does MPICH2 include a remote execution command that is ssh-like? - Steve Weston