Skip to content

Parallel computing: how to transmit multiple parameters to a function in parLapply?

3 messages · Yu Wan, Alexios Ghalanos, William Dunlap

Original
Yu Wan
# 
Hi R-developers

In the package Parallel, the function parLapply(cl, x, f) seems to allow
transmission of only one parameter (x) to the function f. Hence in order to
compute f(x, y) parallelly, I had to define f(x, y) as f(x) and tried to
access y within the function, whereas y was defined outside of f(x).

Script:

library(parallel)

f <- function(x) {
  z <- 2 * x + .GlobalEnv$y  # Try to access y in the global scope.
  return(z)
}

np <- detectCores(logical = FALSE)  # Two cores of my laptop
x <- seq(1, 10, by = 1)
y <- 0.5  # Y may be an array in reality.
cl <- makeCluster(np)  # initiate the cluster
  r <- parLapply(cl, x, f)  # apply f to x for parallel computing
stopCluster(cl)

The r was a list with 10 empty elements which means f failed to access y.

Then I tested f without parallel computing:
z <- f(x)
print(z)
[1]  2.5  4.5  6.5  8.5 10.5 12.5 14.5 16.5 18.5 20.5

The results indicates that we can access y using .GlobalEnv$y in a function
without parLapply.

The question is, is there any method for me to transmit y to f, or access y
within f during parallel computing?

The version of my R is 3.0.1 and I am running it on a Win8-64x system.

Thanks,

Yu



--
View this message in context: http://r.789695.n4.nabble.com/Parallel-computing-how-to-transmit-multiple-parameters-to-a-function-in-parLapply-tp4682667.html
Sent from the R devel mailing list archive at Nabble.com.
Alexios Ghalanos
# 
This works:

clusterExport(cl, c("f","y"), envir=environment())
r <- parLapply(cl, x, function(x) f(x,y))

You need to export your function (?f?) and additional variables (?y?), and then 
define that function inside parLapply ("f(x,y)?). If you were to also make use of
additional libraries (or source some scripts) then you should also consult 
?clusterEvalQ?.
The makeCluster command (at least in windows via socket) just initializes new R 
processes which do not know about your functions or variables unless you
export those to them.

Perhaps a question best suited for R-help.

Alexios
On 24 Dec 2013, at 06:15, Yu Wan <walterwan at 126.com> wrote:

            

      
      
      
    

        

      
      
    

  
  


  


      

    

    

      
      

        


  

        

      William Dunlap
    

    

      
      #
    

  


  

      
You can put the function of interest and any global
variables it needs into a private environment, which gets sent
along with the function to the child processes.  E.g.

library(parallel)
cl3 <- makeCluster(3)
y <- c(1,100,10000)
addY <- function(x) x + y
withGlobals <- function(FUN, ...){
    environment(FUN) <- list2env(list(...))
    FUN
}
parLapply(cl3, 1:4, withGlobals(addY, y=y))
# [[1]]
# [1]     2   101 10001
# 
# [[2]]
# [1]     3   102 10002
# ...

Bill Dunlap
Spotfire, TIBCO Software
wdunlap tibco.com