Hi, I have a C package which includes the usage of the StarPU runtime system to allow running my code on different HW architectures. I included this to an R-package and I found that R is using only one core and all the thread initiated by the runtime system is assigned to a single core which impacted the performance. I used Sys.setenv(KMP_AFFINITY= "disabled") to disable the default affinity and this helps to allow the runtime system to use all available cores. However, even I gained speedup, the R package is still 5X slower than the C package. Could anyone help me to understand the problem better? --Sameh
[R-pkg-devel] Problem with using parallel runtime system with R
7 messages · Sameh M. Abdulah, Iñaki Ucar
On Fri, 10 May 2019 at 13:48, Sameh M. Abdulah
<sameh.abdulah at kaust.edu.sa> wrote:
Hi, I have a C package which includes the usage of the StarPU runtime system to allow running my code on different HW architectures. I included this to an R-package and I found that R is using only one core and all the thread initiated by the runtime system is assigned to a single core which impacted the performance. I used Sys.setenv(KMP_AFFINITY= "disabled") to disable the default affinity and this helps to allow the runtime system to use all available cores. However, even I gained speedup, the R package is still 5X slower than the C package. Could anyone help me to understand the problem better?
Is this package publicly available somewhere? I?aki
Actually not yet. It is on Github but in a private repo. If it helps I can fork it for public access for now. Sameh Get Outlook for iOS<https://aka.ms/o0ukef>
On Fri, May 10, 2019 at 5:55 PM +0300, "I?aki Ucar" <iucar at fedoraproject.org<mailto:iucar at fedoraproject.org>> wrote:
On Fri, 10 May 2019 at 13:48, Sameh M. Abdulah wrote:
Hi, I have a C package which includes the usage of the StarPU runtime system to allow running my code on different HW architectures. I included this to an R-package and I found that R is using only one core and all the thread initiated by the runtime system is assigned to a single core which impacted the performance. I used Sys.setenv(KMP_AFFINITY= "disabled") to disable the default affinity and this helps to allow the runtime system to use all available cores. However, even I gained speedup, the R package is still 5X slower than the C package. Could anyone help me to understand the problem better?
Is this package publicly available somewhere? I?aki
On Fri, 10 May 2019 at 17:06, Sameh M. Abdulah
<sameh.abdulah at kaust.edu.sa> wrote:
Actually not yet. It is on Github but in a private repo. If it helps I can fork it for public access for now.
Providing the code always improves your chances of getting help. Otherwise, the information is probably too limited. I?aki
I will try to talk to my team to fork it for more investigation from your side. However, this is the story. We have a C package which uses a STARPU runtime system (http://starpu.gforge.inria.fr/doc/starpu.pdf) to facility the parallelization through a set of task-based parallel algorithms. Once you execute Your program StarPU created several threads to parallelize your tasks based on a specific scheduling mechanism. In the case of R, R is used only one process and if I used the default behavior without any modification, all the StarPU threads are using the same physical core which is used by the default R process. We added A simple modification which helps to well spawn the threads to the other available physical cores by disabling the KMP_AFFINITY (Sys.setenv(KMP_AFFINITY= "disabled")). However, Even the performance is improved, the R package is about 5X slow up compared to the C version. The difference we found that In case of R different threads is controlled by R which means that we can see from top Command that R is using n threads on n cores. In the case of C package, StarPU is using these threads. One more difference from htop command that the virtual memory usage of the R package is doubled which we cannot Understand or even know if it is related to the performance issue or not --Sameh
?On 5/10/19, 6:12 PM, "I?aki Ucar" <iucar at fedoraproject.org> wrote:
On Fri, 10 May 2019 at 17:06, Sameh M. Abdulah
<sameh.abdulah at kaust.edu.sa> wrote:
>
> Actually not yet. It is on Github but in a private repo. If it helps I can fork it for public access for now.
Providing the code always improves your chances of getting help.
Otherwise, the information is probably too limited.
I?aki
On Fri, 10 May 2019 at 17:38, Sameh M. Abdulah
<sameh.abdulah at kaust.edu.sa> wrote:
I will try to talk to my team to fork it for more investigation from your side.
Disclaimer: I'm not saying that I'll have time to have a look. Just that somebody might help if you provide more information, especially the actual code, beyond stating that it is not working. I?aki
I meant for anyone who can help :). ?Sameh Get Outlook for iOS<https://aka.ms/o0ukef>
On Fri, May 10, 2019 at 6:48 PM +0300, "I?aki Ucar" <iucar at fedoraproject.org<mailto:iucar at fedoraproject.org>> wrote:
On Fri, 10 May 2019 at 17:38, Sameh M. Abdulah wrote:
I will try to talk to my team to fork it for more investigation from your side.
Disclaimer: I'm not saying that I'll have time to have a look. Just that somebody might help if you provide more information, especially the actual code, beyond stating that it is not working. I?aki