Hello,
I have a piece of code that needs parallelization and it used to work just
fine before (about 6 months ago). However, I had to rerun it yesterday and
found out that my code is now behaving in a weird way - not all worker
processes are charged with the computation. I created a little code that
allows reproducing the issue. Here is a snapshot of the top command:
PID COMMAND %CPU TIME
872 R 97.7 00:02.45
871 R 0.0 00:00.03
870 R 98.2 00:02.93
869 R 0.0 00:00.03
868 R 97.7 00:02.46
867 R 0.0 00:00.03
866 R 94.4 00:02.36
862 R 1.0 00:04.64
Interestingly, the mclapply command seems to correctly charge all workers
instantly.
So is this an intended behavior?
Example that reproduces the issue on my machine:
#
rm(list = ls())
library(parallel)
set.seed(123)
#
expr <- expression(pnorm(b0 + b1*x1 + b2*x2 + b3*x3))
G <- list(
D(D(expr, "x2"), "b0"),
D(D(expr, "x2"), "b1"),
D(D(expr, "x2"), "b2"),
D(D(expr, "x2"), "b3"))
#
b0 <- 1.2
b1 <- 0.4
b2 <- 0.2
b3 <- -0.6
x1 <- rnorm(10^7)
x2 <- rnorm(10^7)
x3 <- rnorm(10^7)
#
nc <- detectCores() - 1
cl <- makeCluster(nc, type = "FORK")
grad_g <- parLapply(cl, G, function(Z) lapply(Z, function(x)
mean(eval(x))))
stopCluster(cl)
#
sessionInfo()
R version 3.2.3 (2015-12-10)
Platform: x86_64-apple-darwin13.4.0 (64-bit)
Running under: OS X 10.11.2 (El Capitan)
locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
attached base packages:
[1] parallel stats graphics grDevices utils datasets methods
base
----
Ce message electronique et tous les fichiers attaches qu'il contient sont
confidentiels et destines exclusivement ? l'usage de la personne ? laquelle
ils sont adresses. Si vous avez re?u ce message par erreur, merci de le
retourner ? son metteur. Les idees et opinions presentees dans ce message
sont celles de son auteur, et ne representent pas necessairement celles de
l'institution ou entite affiliee dont l'auteur est l'employe. La
publication, l'usage, la distribution, l'impression ou la copie non
autorisee de ce message et des attachements qu'il contient sont strictement
interdits.
This email and any files transmitted with it are confide...{{dropped:10}}