Problems parallelizing glmnet

y<-rnorm(1000)
x<-matrix(rnorm(1000*10000),ncol=10000)
dimx<-dim(x)

library(doParallel)
library(foreach)
cl <- makeCluster(8, methods=FALSE)
registerDoParallel(cl)
print(system.time(
pval <- foreach (i =1:dimx[2], .combine=c) %dopar% {
mod <- lm(y ~ x[,i])
summary(mod)$coefficients[2,4]
}
))

  user  system elapsed
 12.28    2.75  231.93

stopCluster(cl)

library(parallel)
cl <- makeCluster(8, methods=FALSE)
print(system.time(
pval <- unlist(mclapply(1:dimx[2], function(i) summary(lm(y ~ x[,i]))$coefficients[2,4]))
))

  user  system elapsed
 21.80    1.33   25.78

stopCluster(cl)

"Brian G. Peterson" <brian at braverock.com> 09/06/12 20:03 PM >>>

On 09/06/2012 12:15 PM, Patrik Waldmann wrote:

I would like to avoid foreach since we showed earlier that it is VERY slow.

A search of the list archives doesn't provide any backup that you have
'showed' anything.

The only other post by you that I can see on the list shows the total
elapsed time decreasing with the use of foreach, even though it was,
indeed, a trivial function evaluation.

--
Brian

In this case, each iteration of the function is very quick:

system.time(summary(lm(y ~ x[,1]))$coefficients[2,4])

   user  system elapsed
   0.01    0.00    0.02

And you are doing 10,000 iterations, so overhead matters a lot.  In the
glmnet problem, each iteration of the function is very slow, and you are
doing 8 iterations, so overhead doesn't matter at all.

Finally, I suspect that using the doMC foreach backend will improve things
considerably, but I can't currently test that.

On Thu, Sep 6, 2012 at 1:58 PM, Zachary Mayer <zach.mayer at gmail.com> wrote:

In this case, each iteration of the function is very quick:

system.time(summary(lm(y ~ x[,1]))$coefficients[2,4])

   user  system elapsed
   0.01    0.00    0.02

And you are doing 10,000 iterations, so overhead matters a lot.  In the
glmnet problem, each iteration of the function is very slow, and you are
doing 8 iterations, so overhead doesn't matter at all.

Finally, I suspect that using the doMC foreach backend will improve things
considerably, but I can't currently test that.

FWIW, the foreach construct itself (without any parallel backend) is
quite slow and I would not use to loop over a large number of quick
calculations.

Peter

FWIW, the foreach construct itself (without any parallel backend) is
quite slow and I would not use to loop over a large number of quick
calculations.