SUGGESTION: Settings to disable forked processing in R, e.g. parallel::mclapply()
On 4/15/19 11:02 AM, I?aki Ucar wrote:
On Mon, 15 Apr 2019 at 08:44, Tomas Kalibera <tomas.kalibera at gmail.com> wrote:
On 4/13/19 12:05 PM, I?aki Ucar wrote:
On Sat, 13 Apr 2019 at 03:51, Kevin Ushey <kevinushey at gmail.com> wrote:
I think it's worth saying that mclapply() works as documented
Mostly, yes. But it says nothing about fork's copy-on-write and memory overcommitment, and that this means that it may work nicely or fail spectacularly depending on whether, e.g., you operate on a long vector.
R cannot possibly replicate documentation of the underlying operating systems. It clearly says that fork() is used and readers who may not know what fork() is need to learn it from external sources. Copy-on-write is an elementary property of fork().
Just to be precise, copy-on-write is an optimization widely deployed in most modern *nixes, particularly for the architectures in which R usually runs. But it is not an elementary property; it is not even possible without an MMU.
Yes, old Unix systems without virtual memory had fork eagerly copying. Not relevant today, and certainly not for systems that run R, but indeed people interested in OS internals can look elsewhere for more precise information. Tomas