Skip to content
Prev 1977 / 2152 Next
Original Raw

Differing Behaviors in makeCluster between parallel and snow

Benjamin Nutter 
For the past couple of weeks I've been working on constructing a cluster of
Windows computers into a cluster I can use for simulation studies.  For
now, I'm using my local machine and a single remote machine.  With a great
deal of reading and--admittedly--trial and error, I eventually got a
cluster working using the `snow` package.  I haven't been able to get the
same cluster to work using the `parallel` package, and I'm curious as to
why this is the case.

The details of how I've put together my configuration are on a question on
Stack Overflow (
http://stackoverflow.com/questions/32571996/multi-computer-makepsockcluster-on-windows-building-a-step-by-step-guide
).

After getting all of the configurations on both of the machines set up to
be identical, I can run

library(snow)
cl <- snow::makeCluster(c(rep("localhost", 1), rep("[remote_ip]", 4)),
                  rscript = "RScript",
                  rshcmd = "C:/PuTTY/plink.exe -pw [password]",
                  host = "[local_ip]")

This provides a working cluster.

If I run what I believe to be the equivalent code for the `parallel` package

library(parallel)
cl <- parallel::makeCluster(c(rep("localhost", 1), rep("[remote_ip]", 4)),
                  rscript = "RScript",
                  rshcmd = "C:/PuTTY/plink.exe -pw [password]",
                  master = "[local_ip]")

This results in a single SSH session starting on my remote computer which
immediately terminates, and then the function appears to stop processing.
No errors, warnings, or messages are returned.

Admittedly, I haven't set up silent authentication (I'm having trouble
getting my SSH server to accept private tokens).  So my first question
would be if the lack of silent authentication is the likely cause for
`parallel` not completing the cluster while `snow` is able to complete the
cluster with a password being passed.  This behavior certainly qualifies as
"setups which require a password to be supplied may or may not work." (from
the parallel vignette) so I might be incorrectly anticipating that
`parallel` would tolerate the password when `snow` tolerates it.

Might there be any other details I'm missing in putting together a cluster
using `parallel`?

System Details:
Both computers are running R-3.2.2 on 64-bit Windows 7.
R is installed to the same directory on both systems
The .libPaths() are identical on each system.
The packages installed are nearly identical on each system, with the
exception of a couple of Bioconductor packages.

Many thanks,
Benjamin