[R-pkg-devel] Too many cores used in examples (not caused by data.table)
Thanks, I tried with OMP_NUM_THREADS=2 but unfortunately got the same NOTE as before. Best, Jouni
From: Dirk Eddelbuettel <edd at debian.org>
Sent: Wednesday, October 25, 2023 13:27
To: Dirk Eddelbuettel <edd at debian.org>; Helske, Jouni <jouni.helske at jyu.fi>
Cc: Ivan Krylov <krylov.r00t at gmail.com>; r-package-devel at r-project.org <r-package-devel at r-project.org>
Subject: Re: [R-pkg-devel] Too many cores used in examples (not caused by data.table)
Sent: Wednesday, October 25, 2023 13:27
To: Dirk Eddelbuettel <edd at debian.org>; Helske, Jouni <jouni.helske at jyu.fi>
Cc: Ivan Krylov <krylov.r00t at gmail.com>; r-package-devel at r-project.org <r-package-devel at r-project.org>
Subject: Re: [R-pkg-devel] Too many cores used in examples (not caused by data.table)
On 24 October 2023 at 08:15, Dirk Eddelbuettel wrote:
|
| On 24 October 2023 at 15:55, Ivan Krylov wrote:
| | ? Tue, 24 Oct 2023 10:37:48 +0000
| | "Helske, Jouni" <jouni.helske at jyu.fi> ?????:
| |
| | > Examples with CPU time > 2.5 times elapsed time
| | > user system elapsed ratio
| | > exchange 1.196 0.04 0.159 7.774
| |
| | I've downloaded the archived copy of the package from the CRAN FTP
| | server, installed it and tried:
| |
| | library(bssm)
| | Sys.setenv("OMP_THREAD_LIMIT" = 2)
| | data("exchange")
| | model <- svm(
| | exchange, rho = uniform(0.97,-0.999,0.999),
| | sd_ar = halfnormal(0.175, 2), mu = normal(-0.87, 0, 2)
| | )
| | system.time(particle_smoother(model, particles = 500))
| | # user system elapsed
| | # 0.515 0.000 0.073
| |
| | I set a breakpoint on clone() [*] and got quite a few calls creating
| | OpenMP threads with the following call stack:
| |
| | #0 clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:52
| | <...>
| | #4 0x00007ffff7314e0a in GOMP_parallel () from
| | /usr/lib/x86_64-linux-gnu/libgomp.so.1
| | <-- RcppArmadillo code below
| | #5 0x00007ffff38f5f00 in
| | arma::eglue_core<arma::eglue_div>::apply<arma::Mat<double>,
| | arma::eOp<arma::eOp<arma::Col<double>, arma::eop_exp>,
| | arma::eop_scalar_times>, arma::eOp<arma::eOp<arma::Col<double>,
| | arma::eop_scalar_div_post>, arma::eop_square> > (outP=..., x=...) at
| | .../library/RcppArmadillo/include/armadillo_bits/mp_misc.hpp:69
| | #6 0x00007ffff3a31246 in
| | arma::Mat<double>::operator=<arma::eOp<arma::eOp<arma::Col<double>,
| | arma::eop_exp>, arma::eop_scalar_times>,
| | arma::eOp<arma::eOp<arma::Col<double>, arma::eop_scalar_div_post>,
| | arma::eop_square>, arma::eglue_div> (X=..., this=0x7fffffff36f0) at
| | .../library/RcppArmadillo/include/armadillo_bits/Proxy.hpp:226
| | #7
| | arma::Col<double>::operator=<arma::eGlue<arma::eOp<arma::eOp<arma::Col<double>,
| | arma::eop_exp>, arma::eop_scalar_times>,
| | arma::eOp<arma::eOp<arma::Col<double>, arma::eop_scalar_div_post>,
| | arma::eop_square>, arma::eglue_div> > ( X=..., this=0x7fffffff36f0) at
| | .../library/RcppArmadillo/include/armadillo_bits/Col_meat.hpp:535
| | <-- bssm code below
| | #8 ssm_ung::laplace_iter (this=0x7fffffff15e0, signal=...) at
| | model_ssm_ung.cpp:310
| | #9 0x00007ffff3a36e9e in ssm_ung::approximate (this=0x7fffffff15e0) at
| | .../library/RcppArmadillo/include/armadillo_bits/arrayops_meat.hpp:27
| | #10 0x00007ffff3a3b3d3 in ssm_ung::psi_filter
| | (this=this at entry=0x7fffffff15e0, nsim=nsim at entry=500, alpha=...,
| | weights=..., indices=...) at model_ssm_ung.cpp:517
| | #11 0x00007ffff3948cd7 in psi_smoother (model_=..., nsim=nsim at entry=500,
| | seed=seed at entry=1092825895, model_type=model_type at entry=3) at
| | R_psi.cpp:131
| |
| | What does arma::eglue_core do?
| |
| | (gdb) list
| | /* reformatted a bit */
| | library/RcppArmadillo/include/armadillo_bits/mp_misc.hpp:64
| | int n_threads = (std::min)(
| | int(arma_config::mp_threads),
| | int((std::max)(int(1), int(omp_get_max_threads())))
| | );
| | (gdb) p arma_config::mp_threads
| | $3 = 8
| | (gdb) p (int)omp_get_max_threads()
| | $4 = 16
| | (gdb) p (char*)getenv("OMP_THREAD_LIMIT")
| | $7 = 0x555556576b91 "2"
| | (gdb) p /x (int)omp_get_thread_limit()
| | $9 = 0x7fffffff
| |
| | Sorry for misinforming you about the OMP_THREAD_LIMIT environment
| | variable: the OpenMP specification requires the program to ignore
| | modifications to the environment variables after the program has
| | started [**], so it only works if R is started with OMP_THREAD_LIMIT
| | set. Additionally, the OpenMP thread limit is not supposed to be
| | adjusted at runtime at all [***].
| |
| | Unfortunately for our situation, Armadillo is very insistent in setting
| | its own number of threads from arma_config::mp_threads (which is
| | constexpr 8 unless you set preprocessor directives while compiling it)
| | and omp_get_max_threads (which is the upper bound on the number of
| | threads that cannot be adjusted at runtime).
| |
| | What I'm about to suggest is a terrible hack, but since Armadillo seems
| | to lack the option to set the number of threads at runtime, there might
| | be no other option.
| |
| | Before you #include an Armadillo header, every time:
| |
| | 1. #include <omp.h> so that the OpenMP functions are declared and the
| | #include guard is set
| |
| | 2. Define a static inline function get_number_of_threads returning the
| | desired number of threads as an int (e.g. referencing an extern int
| | number_of_threads stored elsewhere)
| |
| | 3. #define omp_get_max_threads get_number_of_threads
| |
| | Now if you provide an API for the R code to get and set this number, it
| | should be possible to control the number of threads used by OpenMP code
| | in Armadillo. Basically, a data.table::setDTthreads() for the copy of
| | Armadillo inlined inside your package.
| |
| | If you then compile your package with a large #define
| | ARMA_OPENMP_THREADS, it will both be able to use more than 8 threads
| | *and* limit itself when needed.
| |
| | An alternative course of action is compiling your package with #define
| | ARMA_OPENMP_THREADS 2 and giving up on more OpenMP threads inside calls
| | to Armadillo.
|
| We should work on adding such a run-time setter of the number of cores to
| RcppArmadillo so that examples can dial down to 2 cores. I have been doing
| just that in package tiledb (via a setting internal to the TileDB Core
| library) for 'ages' now and RcppArmadillo could and should offer the same.
A run-time setter won't work, Armadillo uses a constexpr. But we may have
looked in error at the _wrong_ OpenMP environment variable. Can you please
try with OMP_NUM_THREADS=2 (instead of OMP_THREAD_LIMIT) ?
Dirk
| Dirk
|
| | --
| | Best regards,
| | Ivan
| |
| | [*]
| | https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftidymodels%2Ftextrecipes%2Fpull%2F251%23issuecomment-1775549814&data=05%7C01%7Cjovetale%40jyu.mail.onmicrosoft.com%7C10f4432f57304393f6aa08dbd5450b40%7Ce9662d58caa44bc1b138c8b1acab5a11%7C1%7C0%7C638338264751478030%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=PmuX0x1eRky1TFSwmq7O2cJwCoS87v72xh0ydzHOsA4%3D&reserved=0<https://github.com/tidymodels/textrecipes/pull/251#issuecomment-1775549814>
| |
| | [**]
| | https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.openmp.org%2Fspec-html%2F5.2%2Fopenmpch21.html%23x432-59000021&data=05%7C01%7Cjovetale%40jyu.mail.onmicrosoft.com%7C10f4432f57304393f6aa08dbd5450b40%7Ce9662d58caa44bc1b138c8b1acab5a11%7C1%7C0%7C638338264751478030%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=cTZKytn22QvL%2FOoP7EOm4bqXG%2FUgke%2BhTNT2Mhd6l0k%3D&reserved=0<https://www.openmp.org/spec-html/5.2/openmpch21.html#x432-59000021>
| |
| | [***]
| | https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.openmp.org%2Fwp-content%2Fuploads%2FOpenMPRefCard-5-2-web.pdf%23page%3D15&data=05%7C01%7Cjovetale%40jyu.mail.onmicrosoft.com%7C10f4432f57304393f6aa08dbd5450b40%7Ce9662d58caa44bc1b138c8b1acab5a11%7C1%7C0%7C638338264751478030%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=vYfueL7%2FvEWcPTQZ2Tjivys2kxEXhyDRCkF0u0IHaPg%3D&reserved=0<https://www.openmp.org/wp-content/uploads/OpenMPRefCard-5-2-web.pdf#page=15>
| |
| | ______________________________________________
| | R-package-devel at r-project.org mailing list
| | https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstat.ethz.ch%2Fmailman%2Flistinfo%2Fr-package-devel&data=05%7C01%7Cjovetale%40jyu.mail.onmicrosoft.com%7C10f4432f57304393f6aa08dbd5450b40%7Ce9662d58caa44bc1b138c8b1acab5a11%7C1%7C0%7C638338264751478030%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=fWW%2BqMTeoQpg3%2BeNw%2Fuptmdy9PXsgHMfVijyTv0pdng%3D&reserved=0<https://stat.ethz.ch/mailman/listinfo/r-package-devel>
|
| --
| dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org
|
| ______________________________________________
| R-package-devel at r-project.org mailing list
| https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstat.ethz.ch%2Fmailman%2Flistinfo%2Fr-package-devel&data=05%7C01%7Cjovetale%40jyu.mail.onmicrosoft.com%7C10f4432f57304393f6aa08dbd5450b40%7Ce9662d58caa44bc1b138c8b1acab5a11%7C1%7C0%7C638338264751478030%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=fWW%2BqMTeoQpg3%2BeNw%2Fuptmdy9PXsgHMfVijyTv0pdng%3D&reserved=0<https://stat.ethz.ch/mailman/listinfo/r-package-devel>
--
dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org