[R-pkg-devel] Too many cores used in examples (not caused by data.table)
Hi Dirk, Actually, the OMP_NUM_THREADS worked for vignettes and testthat tests, but didn't help with the examples. However, I just wrapped the problematic example with \donttest as for some reason this issue only happened with a single seemingly simple example (hence the "weird" in the earlier NEWS due to frustration, I changed this to the CRAN version). Thanks for reminding me about the resetting the number of cores, will fix that to the next version. Best, Jouni
From: Dirk Eddelbuettel <edd at debian.org>
Sent: Friday, October 27, 2023 15:42
To: Helske, Jouni <jouni.helske at jyu.fi>
Cc: Dirk Eddelbuettel <edd at debian.org>; Ivan Krylov <krylov.r00t at gmail.com>; r-package-devel at r-project.org <r-package-devel at r-project.org>; Conrad Sanderson <conradsand at gmail.com>
Subject: Re: [R-pkg-devel] Too many cores used in examples (not caused by data.table)
Sent: Friday, October 27, 2023 15:42
To: Helske, Jouni <jouni.helske at jyu.fi>
Cc: Dirk Eddelbuettel <edd at debian.org>; Ivan Krylov <krylov.r00t at gmail.com>; r-package-devel at r-project.org <r-package-devel at r-project.org>; Conrad Sanderson <conradsand at gmail.com>
Subject: Re: [R-pkg-devel] Too many cores used in examples (not caused by data.table)
Jouni,
My CRANberriesFeed reports a new bssm package at CRAN, congratulations for
sorting this out. [1,2] The OMP_NUM_THREADS setting is indeed all it takes,
and it _does_ seem to be read even from a running session: i.e. you can set
this inside an R session and the OpenMP code considers it in the same
process. Good!
As some of us mentioned, your usage pattern of setting
'Sys.setenv("OMP_NUM_THREADS" = 2)' everywhere _leaves_ that value so you
permanently ham-string the behaviour of a session which runs an example or
test of your package: the same session will never get back to 'all cores' by
itself so adding a resetter to the initial value (maybe via on.exit()) may be
a good idea for the next package revision if you have any energy left for
this question :)
Again, congrats for sorting it out, and sorry for the trouble. I long argued
CRAN should set the behaviour-defining environment variable, that
OMP_NUM_THREADS, for the tests and examples it wants to run under reduced
load. Alas, that's not where we ended up.
Cheers, Dirk
[1] https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdirk.eddelbuettel.com%2Fcranberries%2F2023%2F10%2F27%23bssm_2.0.2&data=05%7C01%7Cjovetale%40jyu.mail.onmicrosoft.com%7Caf8b71968c244eb46b6f08dbd6ea3439%7Ce9662d58caa44bc1b138c8b1acab5a11%7C1%7C0%7C638340073635955909%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=mokRvujnBW6XNQPi3fSw6MNJt3Qi2BgNGRwF3dlzNyo%3D&reserved=0<http://dirk.eddelbuettel.com/cranberries/2023/10/27#bssm_2.0.2>
[2] Your NEWS file calls this 'fix weird CRAN issues with parallelisation on
Debian.'. There is nothing 'weird' here (it behaves as designed, computers do
that to us), and it is not just on Debian but on any system where the build
has a) access to OpenMP so uses it and b) measures real time to elapsed time
with a cap of 2 as CRAN does.
--
dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org