mclapply() hangs when keras-based neural networks are involved

Marius,

Tensorflow doesn?t support any parallel computing including forking. It is assumed that all parallelization is done by TF itself and it takes over all resources in a way such that they cannot be shared across processes. Hence you cannot combine TF and parallel (and hence by induction Keras).

Cheers,
Simon

On Aug 28, 2019, at 3:32 AM, Marius Hofert <marius.hofert at uwaterloo.ca> wrote:

Hi,

Below is an example where mclapply() 'hangs' after starting the work
on two cores.
This happens on macOS and Ubuntu (sessionInfo() below). I also see no activity
on 'htop'. lapply() works, though. What is the cause of this behavior?

Cheers,
M

library(tensorflow)
library(keras)
library(parallel)
## TensorFlow also needs to be installed, which can be done via
install_tensorflow() from R

## 1) Setup
in.lay <- layer_input(shape = 2)
hid.lay <- layer_dense(in.lay,  units = 300, activation = "relu")
out.lay <- layer_dense(hid.lay, units = 2,   activation = "sigmoid")
NN <- keras_model(in.lay, out.lay)
loss_fn <- function(x, y = out.lay) loss_mean_squared_error(x, y)
NN %>% compile(optimizer = "adam", loss = loss_fn)

## 2) Training
NN %>% fit(x = matrix(runif(10000 * 2), ncol = 2), # prior data
          y = matrix(rnorm(10000 * 2), ncol = 2), # training data
          batch_size = 5000, epochs = 1)

## 3) Generate samples by evaluating the NN on a prior sample
aux <- function(b) {
   cat(paste("Working on case",b,"\n"))
   Sys.sleep(2)
   predict(NN, x = matrix(runif(100 * 2), ncol = 2)) # mclapply()
hangs here (on macOS and Ubuntu)
}

## 4) Call that hangs after the two processes are started
res.serial   <-   lapply(1:5, function(b) aux(b)) # works
res.parallel <- mclapply(1:5, function(b) aux(b), mc.cores = 2) #
hangs once both cores are used

## Output:
## For lapply():
Working on case 1
Working on case 2
Working on case 3
Working on case 4
Working on case 5
## For mclapply():
Working on case 1
Working on case 2

## sessionInfo() on macOS:
R version 3.6.1 (2019-07-05)
Platform: x86_64-apple-darwin18.7.0 (64-bit)
Running under: macOS Mojave 10.14.6

Matrix products: default
BLAS:   /usr/local/R/R-3.6.1_build/lib/libRblas.dylib
LAPACK: /usr/local/R/R-3.6.1_build/lib/libRlapack.dylib

locale:
[1] en_CA.UTF-8/en_CA.UTF-8/en_CA.UTF-8/C/en_CA.UTF-8/en_CA.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

loaded via a namespace (and not attached):
[1] compiler_3.6.1 tools_3.6.1

## sessionInfo() on Ubuntu:
R version 3.6.0 (2019-04-26)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.3 LTS

Matrix products: default
BLAS:   /u/mhofert/soft/R/R-3.6.0_build/lib/libRblas.so
LAPACK: /u/mhofert/soft/R/R-3.6.0_build/lib/libRlapack.so

locale:
[1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8       LC_NAME=C
[9] LC_ADDRESS=C               LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

loaded via a namespace (and not attached):
[1] compiler_3.6.0