Message-ID: <6C215610-CA71-48E6-937C-453823507823@dcn.davis.ca.us>
Date: 2020-07-08T04:40:50Z
From: Jeff Newmiller
Subject: parallel number of cores according to memory?
In-Reply-To: <CAPr7RtUd99n_VeYG0AFnBe5ZfgprgZWuW4qgdNSLvgS+9guJxQ@mail.gmail.com>
Use an operating system that supports forking, like Linux or MacOSX, and use the parallel package mclapply function or similar to share memory for read operations. [1]
And stop posting in HTML here.
[1] https://cran.r-project.org/web/views/HighPerformanceComputing.html
On July 7, 2020 9:20:39 PM PDT, ivo welch <ivowel at gmail.com> wrote:
>if I understand correctly, R makes a copy of the full environment for
>each
>process. thus, even if I have 32 processors, if I only have 64GB of
>RAM
>and my R process holds about 10GB, I should probably not spawn 32
>processes.
>
>has anyone written a function that sets the number of cores for use (in
>mclapply) to be guessed at by appropriate memory requirements (e.g.,
>"amount-of-RAM"/"RAM held by R")?
>
>(it would be even nicer if I could declare my 8GB data frame to be
>read-only and to be shared among my processes, but this is presumably
>technically very difficult.)
>
>pointers appreciated.
>
>/iaw
>
> [[alternative HTML version deleted]]
>
>______________________________________________
>R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
>https://stat.ethz.ch/mailman/listinfo/r-help
>PLEASE do read the posting guide
>http://www.R-project.org/posting-guide.html
>and provide commented, minimal, self-contained, reproducible code.
--
Sent from my phone. Please excuse my brevity.