Skip to content

problem with opening more than one SOCK cluster with package snow

4 messages · Jannis, Stephen Weston

Original
Jannis
# 
Der HPC R users,


sorry for cross posting (to r-help) but I just realized after posting 
that the hpc list may be more suitable for my request. I will report 
solutions to both lists.


Cheers
Jannis


Dear r-help list,

i have some problems using the snow package to create a SOCK cluster. 
The errors just occour irregularly but it seems to me that they occour 
when I try to create more than one cluster on the same machine via 
different R instances started via submitting LSF jobs to a cluster. Does 
anyone have an idea how to solve this or where to start digging for 
solutions?

The error messages are:

library(snow)
cl <- makeCluster(8, type = "SOCK")


Error in socketConnection(port = port, server = TRUE, blocking = TRUE,  :
   cannot open the connection
Calls: run.call ... makeCluster -> makeSOCKcluster -> newSOCKnode -> 
socketConnection
In addition: Warning message:
In socketConnection(port = port, server = TRUE, blocking = TRUE,  :
   port 10187 cannot be opened


R version 2.15.1 (2012-06-22)
Platform: x86_64-unknown-linux-gnu (64-bit)

locale:
  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
  [7] LC_PAPER=C                 LC_NAME=C
  [9] LC_ADDRESS=C               LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats     graphics  grDevices datasets  utils     methods   base

other attached packages:
[1] snow_0.3-10


Thanks a lot
Jannis
Stephen Weston
# 
Hi Jannis,

I think you are right as to the cause of the error.  I believe that
the problem occurs when two master processes start running
on the same machine at about the same time because they
will both try to bind to the same port by default.  Interestingly,
the master process only binds to that port temporarily, which
is why the problem is intermittent.  The more workers that you
start, the longer it takes to create the cluster, and the more
likely the error is to occur.

I suggest that you use the "port" option when calling makeCluster,
as in:

  cl <- makeCluster(8, type="SOCK", port=10188)

where you specify a different value for every master on the
same machine.  I'm not aware of a fool-proof technique for
doing that, but you can probably come up with some method
that works pretty well for your situation.

Of course, if you're not using Windows, you could try using
the "parallel" package, and create a "FORK" cluster.  That
doesn't use sockets at all, so multiple clusters on the same
machine isn't a problem.

- Steve
On Thu, Dec 20, 2012 at 7:11 AM, Jannis <bt_jannis at yahoo.de> wrote:
Jannis
# 
Hi Steve,

thanks a lot for your reply. There may be an elegant and fail proof 
method for port determination but I learned during my work that 
sometimes the quick and dirty way has a much higher output/programming 
energy ratio ;-). So I would start with assigning random ports each time 
a cluster is started and restart the odd jobs with conflicting ports. I 
am no expert in ports and related things, but are there possible 
problems with other programs or even security that might occour if I use 
random port numbers between 1023 and 65535?


Thanks a lot
Jannis
On 20.12.2012 16:00, Stephen Weston wrote:
Stephen Weston
# 
You might want to talk to the sysadmins of your cluster to
see what they suggest.  If you start snow clusters in this way
using random ports, the workers that are started may trigger
a lot of error messages in the system log files which may
alarm a watchful sysadmin.

Personally, I would use the multicore package in this situation.

- Steve
On Fri, Dec 21, 2012 at 6:38 AM, Jannis <bt_jannis at yahoo.de> wrote: