[R-pkg-devel] Unexpected multi-core CPU usage in package tests
On Wed, 28 Feb 2024 at 17:36, Ivan Krylov <ikrylov at disroot.org> wrote:
? Tue, 27 Feb 2024 11:14:19 +0000 Jon Clayden <jon.clayden at gmail.com> ?????:
My testing route is to install the packages within the 'rocker/r-devel' Docker container, which is Debian-based, then use 'time' to evaluate CPU usage. Note that, even though 'RNifti' does not use OpenMP, setting OMP_NUM_THREADS changes its CPU usage
I think that's because rocker/r-devel uses parallel OpenBLAS: $ podman run --rm -it docker.io/rocker/r-devel \ R -q -s -e 'sessionInfo()' | grep -A1 BLAS BLAS: /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3 LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.24.so; LAPACK version 3.11.0 The incoming CRAN check machine either sets the BLAS parallellism to 1 or uses a non-parallel BLAS. With rocker/r-devel, you can run R with the environment variable OPENBLAS_NUM_THREADS set to 1. It's been effective in the past to run R -d gdb and set a breakpoint on pthread_create before launching the test. (In theory, it may be required to set a breakpoint on every system call that may be used to create threads, including various variations of clone(), subject to variations between operating systems, but pthread_create has been enough for me so far.) With OPENBLAS_NUM_THREADS=1, I'm only seeing OpenMP threads created by the mmand package during tests for your package tractor.base, and the latest commit (that temporary disables testing of mmand) doesn't hit the breakpoint or raise any NOTEs at all.
Very many thanks for testing this, Ivan. I did come across the breakpoint suggestion when searching on this, and tried it, but I ran into some problems with gdb within the container. Your experiments confirm that my yardstick was just misleading me; I will add support for explicitly setting the thread count within mmand, which should give me a longer-term solution. Thanks again, Jon