Creating different working directories for each node?

To Brian and anyone else following along on this thread:

I wanted to reply with a solution. (Thanks also to Luke Tierney for
feedback.)

The simplest solution seems to be to take advantage of the temporary
directories automatically populated by R / snow / snowfall when the
cluster is initialized and then use these directories to store the
temporary files. Because each tempdir is unique to each process, each
worker by definition is assigned its own tempdir.

In my case, I just needed to add the following into my function:

testfun <- function(...) {
     # run normal code [snipped out here]

     # switch to temporary directory for function that writes/scans temp
file
     wd <- getwd()
     td <- tempdir()
     setwd(td)
     out <- filemakingfunction(...)

     # return to normal working directory
     setwd(wd)

     return(out)
     }

This seems to work without any errors when called within clusterApply()
and clusterApplyLB().

The fix is confirmed by the following: If 'td' above is assigned the
shared 'wd' instead of 'tempdir()', I get nonsensical results for the
portions of the analyses using the temp file.

Thanks again to everyone who provided help!

Phil

Creating different working directories for each node?

Thread (4 messages)