Creating different working directories for each node?
The tempfile option might work, as might using Sys.info()["nodename"] to specify individual subdirectories. Are you aware of any functions that can be used to identify which CPU a process has been spawned to? The package 'snowfall' has sfGetCluster(), but that doesn't seem to be able to be called *within* a cluster function. Thanks again, Phil
On 8/11/2014 4:10 PM, Brian G. Peterson wrote:
You didn't tell us which clusterning mechanism you're using, but most will allow you to retrieve a unique node id or node number, which you could use to create a subdirectory. Also, 'tempfile' in R will create a vector of unique temporary file names, which you could pass to the nodes, or append with a node id. Brian On 08/11/2014 03:59 PM, Novack-Gottshall, Philip M. wrote:
Greetings, I'm trying to run some code on a cluster in which an internal qhull convex-hull function repeatedly writes then scans a dummy file. (Very inefficient, I know, but c'est la vie.) The problem is that all nodes share the same default working directory, and get confused because they are all trying to read/write the same dummy file. (So far as I know, there is no way to specify unique file names for the dummy files in the internal .C function, especially with my lack of C prowess.) I've been playing around with having my function (called using snowfall's 'sfClusterApply') specify unique working directories (using 'setwd'), say one wd for each CPU, so that the individual dummy files are set within unique directories. This seems a plausible wrap-around so long as I confirm that I'm matching up the wd with the correct CPU process. I wanted to check whether anyone has any other recommendations before I waste my time on further troubleshooting. If it's relevant, I'm running my code using management package snowfall on a CentOS/OpenMPI Intel cluster with 2 hyperthreaded 6-cores in each of 16 nodes (allowing 382 functional CPUs/unique working directories for the job). Thanks, Phil
_______________________________________________ R-sig-hpc mailing list R-sig-hpc at r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-hpc
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Phil Novack-Gottshall Associate Professor Department of Biological Sciences Benedictine University 5700 College Road Lisle, IL 60532 pnovack-gottshall at ben.edu Phone: 630-829-6514 Fax: 630-829-6547 Office: 332 Birck Hall Lab: 107 Birck Hall http://www1.ben.edu/faculty/pnovack-gottshall ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~