[R-HPC] Cores hang when calling mcapply
Regarding RAM restrictions, here are the specs:
total used free shared buff/cache available
Mem: 753Gi 70Gi 600Gi 2.9Gi 89Gi 683Gi
Swap: 4.0Gi 2.5Gi 1.5Gi
It's a multi-user server so naturally things fluctuate.
Regarding CPU restrictions, here are the specs of our server:
Thread(s) per core: 2
Core(s) per socket: 20
Socket(s): 4
Stepping: 4
CPU(s) scaling MHz: 50%
CPU max MHz: 3700.0000
CPU min MHz: 1000.0000
Now that I'm looking at it I may be reserving too many in the call so I will scale it back and see if that helps as well.
From: Andrew Ekstrom <aekstrom at umich.edu>
Sent: Wednesday, December 11, 2024 9:12 AM
To: Deramus, Thomas Patrick <tderamus at mgb.org>; r-sig-hpc at r-project.org <r-sig-hpc at r-project.org>
Subject: Re: [R-HPC] Cores hang when calling mcapply
Sent: Wednesday, December 11, 2024 9:12 AM
To: Deramus, Thomas Patrick <tderamus at mgb.org>; r-sig-hpc at r-project.org <r-sig-hpc at r-project.org>
Subject: Re: [R-HPC] Cores hang when calling mcapply
External Email - Use Caution
I got errors like you show before. How much ram do you have access to on this server?
My solution was to come up with a clean up line of code to remove old data from ram AND add an extra 256GB of ram.
Drew Ekstrom
Analytics Expert and Math Lecturer
________________________________
From: R-sig-hpc <r-sig-hpc-bounces at r-project.org> on behalf of Deramus, Thomas Patrick <tderamus at mgb.org>
Sent: Wednesday, December 11, 2024 8:51:00 AM
To: r-sig-hpc at r-project.org <r-sig-hpc at r-project.org>
Subject: Re: [R-sig-hpc] [R-HPC] Cores hang when calling mcapply
Follow up, this is the error I typically get:
Lost warning messages
Error: no more error handlers available (recursive errors?); invoking 'abort' restart
Execution halted
Error: cons memory exhausted (limit reached?)
Error: cons memory exhausted (limit reached?)
Error: cons memory exhausted (limit reached?)
Error: cons memory exhausted (limit reached?)
________________________________
From: R-sig-hpc <r-sig-hpc-bounces at r-project.org> on behalf of Deramus, Thomas Patrick <tderamus at mgb.org>
Sent: Wednesday, December 11, 2024 8:36 AM
To: r-sig-hpc at r-project.org <r-sig-hpc at r-project.org>
Subject: [R-sig-hpc] [R-HPC] Cores hang when calling mcapply
External Email - Use Caution
Hi R-HPC users.
I asked this on the main R-help list so apologies if you're seeing this again, but it was suggested I ask here as well
Additional apologies for the lack of concrete examples because the dataset is large, and it being so I believe is the issue.
I multiple, very large datasets for which I need to generate 0/1 absence/presence columns
Some include over 200M rows, with two columns that need presence/absence columns based on the strings contained within them, as an example, one set has ~29k unique values and the other with ~15k unique values (no overlap across the two).
Using a combination of custom functions:
crewjanitormakeclean <- function(df,columns) {
df <- df |> mutate(across(columns, ~make_clean_names(., allow_dupes = TRUE)))
return(df)
}
mass_pivot_wider <- function(df,column,prefix) {
df <- df |> distinct() |> mutate(n = 1) |> pivot_wider(names_from = glue("{column}"), values_from = n, names_prefix = prefix, values_fill = list(n = 0))
return(df)
}
sum_group_function <- function(df) {
df <- df |> group_by(ID_Key) |> summarise(across(c(starts_with("column1_name_"),starts_with("column2_name_"),), ~ sum(.x, na.rm = TRUE))) |> ungroup()
return(df)
}
and splitting up the data into a list of 110k individual dataframes based on Key_ID
temp <-
open_dataset(
sources = input_files,
format = 'csv',
unify_schema = TRUE,
col_types = schema(
"ID_Key" = string(),
"column1" = string(),
"column1" = string()
)
) |> as_tibble()
keeptabs <- split(temp, temp$ID_Key)
I used a multicore framework to distribute the `sum` functions across each Key_ID when a multicore argument is enabled.
??????
if(isTRUE(multicore)){
output <- mclapply(1:length(modtabs), function(i) crewjanitormakeclean(modtabs[[i]],c("string_columns_2","string_columns_1")), mc.cores = numcores)
output <- mclapply(1:length(modtabs), function(i) mass_pivot_wider(modtabs[[i]],"string_columns_1","col_1_name_"), mc.cores = numcores)
output <- mclapply(1:length(modtabs), function(i) mass_pivot_wider(modtabs[[i]],"string_columns_2","col_2_name_"), mc.cores = numcores)
}else{
output <- lapply(1:length(modtabs), function(i) crewjanitormakeclean(modtabs[[i]],c("string_columns_2","string_columns_1")))
output <- lapply(1:length(modtabs), function(i) mass_pivot_wider(modtabs[[i]],"string_columns_1","col_1_name_"))
output <- mclapply(1:length(modtabs), function(i) mass_pivot_wider(modtabs[[i]],"string_columns_2","col_2_name_"))
}
Moving every Key_ID to a single row and then row-binding the data while creating new columns for the differences across `Key_ID`s from the pivot using the following solution (78 upvotes at time of this email):
https://secure-web.cisco.com/1G1n4sFquFSAGXcAhqOo3c3lp6ipL3mpWrMcrJGlTpwl0u26UB5NiqsLAnhEOsrwLwZGKH9CM-7Xl2iReOmHOWwkbqfaa34jRl2zj6NG6ZPvF1Ev5GA4Yrquq9rXWECDdRRhl0d5KwSNbuM2pc8L6yTNU4MXRsptY-nQdIsDm_nqsvxEFKUE7Za9eyrt0_vkcGMQ596el4L50eQStz1JOCXuAjvKliwISQufp1-lyFdAkY-7MdIw59JGOY1oraDzYl937Vt-XX2r7Dx-v5KsF8VtyB0iMr_wQd5UA-tG8ZwdR8Q4pLKuoUuImGzteFCTK/https%3A%2F%2Fstackoverflow.com%2Fquestions%2F3402371%2Fcombine-two-data-frames-by-rows-rbind-when-they-have-different-sets-of-columns
allNms <- unique(unlist(lapply(keeptabs, names)))
output <- do.call(rbind,
c(lapply(keeptabs,
function(x) data.frame(c((x), sapply(setdiff(allNms, names(x)),
function(y) NA))) |> as_tibble()),
make.row.names=FALSE)) |> mutate(across(c(starts_with("column1_name_"), starts_with("column2_name_")), coalesce, 0))
However, I have noticed that the jobs seem to "hang" after a while, with the initial 30 or so (numcores == 30 in the workflow, equal to the number of cores I reserved) at 100% of the requested CPU, and then several "zombie" processes occur and the cores just stop at 0% and never proceed, usually dying with a timeout or not all jobs running to completion failure to join of some kind.
This happens in both base R and RStudio, and I haven't been able to figure out if it's something wrong with the code, the size of the data, or our architecture, but I would appreciate any suggestions as to what I might be able to do about this.
Before they are suggested, I have also tried this same approach with foreach, snow, future, and furr packages, and base parallel with mc.apply seems to be the only thing that works for at least one dataset.
In the event it has something to do with our architecture, here is what we are running on and our loaded packages:
OS Information:
NAME="Red Hat Enterprise Linux"
VERSION="9.3 (Plow)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="9.3"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Red Hat Enterprise Linux 9.3 (Plow)"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos"
HOME_URL="https://secure-web.cisco.com/1h7_cGThR4E8jHCYamL4o_FHW4nl7BFKMGMRVLQDdaZ0qWJ7A-AO5GfJfswTgIA64F6ti1IIn7r-uj5jO4fxVgTLzVnkMwvZAparKYNGPsv9N7h7nptYFHuoPa6ubPOWa-ptnh5DQr9JWYOwHrtJlvXZIUvxn6htl9SXhlBrJZPtfrD22_uTeb_TC1RG_qy1NjW7Mzye4iE86WIkQnA3bvCCFRBlHWGuBgOlFJgtA1pV3ikrH5CuKkFnUk6igeJp6qoszmYKA2mM-Z67tV7KU0rV4sFfBN4aJJcASSV7a7B0pYa3vwvFovlNsFgq5yTnY/https%3A%2F%2Fwww.redhat.com%2F"
DOCUMENTATION_URL="https://secure-web.cisco.com/1iWVumbt8ShswsWwWdSlpe9J8uvS5O_E5XF6SB08JYI2uBK0xwtlD_JkkSfg6D4XxoxmCj2zyMixuhtA6S8Xh-92U9udozSWm1efT79xD8hDmGxbK_qO4ZpGBu-5YjBRKom4_nQOl5SKgD5HQLS7xg9UPaeKYn94_CWiMScnd9BWHoquMx5XP8Ex-NLoBd0rElgWPFfx77yy6M8QWJ9yBsAc8WJD4E3Zt1oyt9CtOR52NGkG_YvLrxJrNqfluWrfFW6UJAzCX-zgH5JenNpyokC0Aq5Yce88oIbjyCsWISAZ7gMTCdWf1sHwS4wWj2M3M/https%3A%2F%2Faccess.redhat.com%2Fdocumentation%2Fen-us%2Fred_hat_enterprise_linux%2F9"
BUG_REPORT_URL="https://secure-web.cisco.com/1fNMHII_7nlcKfcWImleG3ZP7DT9eRQNmTSf3AkBzirmzCzp1zT3xBlJCfmy1hlrL0Rcn1Oz0DvMTU1qN_gRn4YRxxB7oSgAi7I1HJ6pMa76wdAyN7CHX7vqqS_U4ocSCDvUID48ibjS6kD7sKiwVZA0CnG5EJspkGrhwk6dlxxoeS0mrR1GkOF5CQ8VCfXzYwm80u_RLiLSNmb_udzjvNxHHXRbXupCU2goONv4sADdrVJeM7cxzDa30eRwSwC2F7hvsz0qmVU3rq3p5mNC376QI0ZgItffJ90fFP5uUxVl56Vs0XhD06muVstfxkVdl/https%3A%2F%2Fbugzilla.redhat.com%2F"
REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_BUGZILLA_PRODUCT_VERSION=9.3
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.3"
Operating System: Red Hat Enterprise Linux 9.3 (Plow)
CPE OS Name: cpe:/o:redhat:enterprise_linux:9::baseos
Kernel: Linux 5.14.0-362.13.1.el9_3.x86_64
Architecture: x86-64
Hardware Vendor: Dell Inc.
Hardware Model: PowerEdge R840
Firmware Version: 2.15.1
R Version:
R.Version()
$platform
[1] "x86_64-pc-linux-gnu"
$arch
[1] "x86_64"
$os
[1] "linux-gnu"
$system
[1] "x86_64, linux-gnu"
$status
[1] ""
$major
[1] "4"
$minor
[1] "3.2"
$year
[1] "2023"
$month
[1] "10"
$day
[1] "31"
$`svn rev`
[1] "85441"
$language
[1] "R"
$version.string
[1] "R version 4.3.2 (2023-10-31)"
$nickname
[1] "Eye Holes"
RStudio Server Version:
RStudio 2023.09.1+494 "Desert Sunflower" Release (cd7011dce393115d3a7c3db799dda4b1c7e88711, 2023-10-16) for RHEL 9
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36
PPM Repo:
https://secure-web.cisco.com/1nMuCOnANj6fZMyl2-XZciqjw-DMAGy2kqdTnq_URbhzu9NQeVLtyJW9XYi-aDbPvK7JTLcN0pdEzLY7StrAcbkvQdfn9TbfGA9xdAaGuabqNLUHc6WQP7Lk5es-wAKt2tEf0GfTf4gkg6Lx22c2Xqjb-qUryZ5Fq_0XU2o-fmM9WK8CfLi352XNSlsW6M_sPP57zBgpSUc3vyy97jNOXMcw6f2pYhisuYLA9-EsIM-KQWohyQ_K83SKZ2wuKxRZ1vMEoRTUdOePklEGnQ2c7xBj-zxh5BxQ2I1jxs1pLC3wVq5iRlEJb-2TXB0knuHJD/https%3A%2F%2Fpackagemanager.posit.co%2Fcran%2F__linux__%2Frhel9%2Flatest
attached base packages:
[1] parallel stats graphics grDevices datasets utils methods base
other attached packages:
[1] listenv_0.9.1 microbenchmark_1.5.0 dbplyr_2.4.0 duckplyr_0.4.1 readxl_1.4.3 fastDummies_1.7.3
[7] glue_1.8.0 arrow_14.0.2.1 data.table_1.15.2 toolbox_0.1.1 janitor_2.2.0 lubridate_1.9.3
[13] forcats_1.0.0 stringr_1.5.1 dplyr_1.1.4 purrr_1.0.2 readr_2.1.5 tidyr_1.3.1
[19] tibble_3.2.1 ggplot2_3.5.0 tidyverse_2.0.0 duckdb_1.1.2 DBI_1.2.3 fs_1.6.3
Happy to provide additional information if it would be helpful.
Thank you in advance!
The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Mass General Brigham Compliance HelpLine at https://secure-web.cisco.com/1nHX6LSUt1uyh1BmE4LclbuxPwC8OddfA2wx3IsRTTTsz9DZbYLx4LflIk1fdP-1F34abcDhI9UY45XLj8mqFs_V9rK8svOzFnEPYtIhBqK1Dt-5iY4Z7SWL3eLS-kP-E-vRaWX976TCY8vWOs3xF8Rajk4yWyeox0iOTbl0io5NKXTG-3l2I8QWlqeTdM6eVHOCOAm9f46Ns4DLJJXEkbGSofOk9nWHw1xEiycZHTjU22ka29HpZtgkPT75eiJYpNJrVj-n3tzRSdceEY8pJLusWFS45f-eG_I4IuvlC40HGkyvsZQBJwn0ZzRxTxXK2/https%3A%2F%2Fwww.massgeneralbrigham.org%2Fcomplianceline <https://secure-web.cisco.com/1nHX6LSUt1uyh1BmE4LclbuxPwC8OddfA2wx3IsRTTTsz9DZbYLx4LflIk1fdP-1F34abcDhI9UY45XLj8mqFs_V9rK8svOzFnEPYtIhBqK1Dt-5iY4Z7SWL3eLS-kP-E-vRaWX976TCY8vWOs3xF8Rajk4yWyeox0iOTbl0io5NKXTG-3l2I8QWlqeTdM6eVHOCOAm9f46Ns4DLJJXEkbGSofOk9nWHw1xEiycZHTjU22ka29HpZtgkPT75eiJYpNJrVj-n3tzRSdceEY8pJLusWFS45f-eG_I4IuvlC40HGkyvsZQBJwn0ZzRxTxXK2/https%3A%2F%2Fwww.massgeneralbrigham.org%2Fcomplianceline> .
Please note that this e-mail is not secure (encrypted). If you do not wish to continue communication over unencrypted e-mail, please notify the sender of this message immediately. Continuing to send or respond to e-mail after receiving this message means you understand and accept this risk and wish to continue to communicate over unencrypted e-mail.
[[alternative HTML version deleted]]
_______________________________________________
R-sig-hpc mailing list
R-sig-hpc at r-project.org
https://secure-web.cisco.com/1bNUJMn1Oaz0nrJBQTfSeQ5NQQFhgE-Iu1U5J_ORrurkQqW0owPqGqv0ivpDb3OZwZVSq4venw3mXyOy3skXniKPl_XzmrroHyksT3pqyQVELcmL191rlKsyLlkJLSyAZRMNlvX_4iDfO4AL_sThOzKz3LRbh09pnaqQF_NFmz2xcybe-9h1G0VEbmKvKYh_01SIe9AK6R0o4JxFuH3kFjYzKz395gIGKf8HBN3Uw5nBksh5JcK7fCjFFgv1pY_Pg3iyf6cA35UKbCEiT1kyoRy_BT_S74CtHxfbJ3tkSEHETSXabTgZlzTfcegmMwZG7/https%3A%2F%2Fstat.ethz.ch%2Fmailman%2Flistinfo%2Fr-sig-hpc
The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Mass General Brigham Compliance HelpLine at https://www.massgeneralbrigham.org/complianceline<https://secure-web.cisco.com/1L1ZYrZr5D1EtARKPjvrOrnvQRHBTXEYVqYm5yUPgvjY4zVVzYrY5te0eP0Q3iLSTA500FIBIzm0I7qrKf4OnLPUDnkIjXdyX-GyPZqm26WULrZG7Qdb-yWFynH3dZJz2DrlbZO3SDGL0jcAqYH-nyu0RzT0c_6begHRR46As2ykL7YgVIELpRPUU9vdfnwlmpEI6X_1hGMqoIDyIYIj7VhrAxHhEb5Z--S588TAeSLWUxn4amwzynB7zfLkwj_gKMbxsU1Og-EbIxNfYxotyhV1NzrgZFJ9nBNWMcdt-_jCFif4XlFk9JYUXXDXGto_O/https%3A%2F%2Fwww.massgeneralbrigham.org%2Fcomplianceline> <https://www.massgeneralbrigham.org/complianceline<https://secure-web.cisco.com/1L1ZYrZr5D1EtARKPjvrOrnvQRHBTXEYVqYm5yUPgvjY4zVVzYrY5te0eP0Q3iLSTA500FIBIzm0I7qrKf4OnLPUDnkIjXdyX-GyPZqm26WULrZG7Qdb-yWFynH3dZJz2DrlbZO3SDGL0jcAqYH-nyu0RzT0c_6begHRR46As2ykL7YgVIELpRPUU9vdfnwlmpEI6X_1hGMqoIDyIYIj7VhrAxHhEb5Z--S588TAeSLWUxn4amwzynB7zfLkwj_gKMbxsU1Og-EbIxNfYxotyhV1NzrgZFJ9nBNWMcdt-_jCFif4XlFk9JYUXXDXGto_O/https%3A%2F%2Fwww.massgeneralbrigham.org%2Fcomplianceline>> .
Please note that this e-mail is not secure (encrypted). If you do not wish to continue communication over unencrypted e-mail, please notify the sender of this message immediately. Continuing to send or respond to e-mail after receiving this message means you understand and accept this risk and wish to continue to communicate over unencrypted e-mail.
[[alternative HTML version deleted]]
_______________________________________________
R-sig-hpc mailing list
R-sig-hpc at r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-hpc<https://secure-web.cisco.com/16dvBAQ_1nM6rFujNDuY789zUZ5q8kvB6jba4V7nIawE90qgS2nDWmNU48c_zyGC9DUl42C86-3OhVqHhRnIhJ2xNUrXr11m_xpThL6jdJkyXbFlevDWWyGU0Fy0IuR_8H93ftQUuMT6PcNNF6ee7DH_0n_WbPnjckz-4BxjbXcxQHaOFNi3AZG9MVu8Z69-P0c7MdISr5T5CmYEI8GU0rIyPD81S7maFzR3qkL4IAyoX8OjjI9mn2T8f-5K3xxTOkmd7GIwmANrGqDGyaHF9P6RbllyeQAeYMMYNaMsKHnhMmKOtOm5-mDQPEh2tyc9q/https%3A%2F%2Fstat.ethz.ch%2Fmailman%2Flistinfo%2Fr-sig-hpc>
The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Mass General Brigham Compliance HelpLine at https://www.massgeneralbrigham.org/complianceline <https://www.massgeneralbrigham.org/complianceline> .
Please note that this e-mail is not secure (encrypted). If you do not wish to continue communication over unencrypted e-mail, please notify the sender of this message immediately. Continuing to send or respond to e-mail after receiving this message means you understand and accept this risk and wish to continue to communicate over unencrypted e-mail.