Rsge combined with mclapply?
On Thu, Jun 14, 2012 at 4:42 AM, Daniel Saban?s Bov?
<daniel.sabanesbove at ifspm.uzh.ch> wrote:
Hi Kasper, thank you for this hint! I was not aware of this possibility. Yes, indeed, then it should be trivial to let multiple jobs run on cores of the same node. In the meantime I have found a solution by tweaking the Rsge package to use mclapply() from the base R-package "parallel". This also works.
This might work in the sense that it produces workable code on your system. But are you sure that you do not over-use your resources. For example, on our system, I can qsub a job that SGE thinks takes up a single slot (= core, the default) and then still use mclapply over 8 cores. In that situation, I have asked for just 1 core, but I am really using 8. This has clear implications for the other users on the system as well as resource allocation. This is because our SGE system is not too tightly locked down - it may be possible for a sysadmin to prevent this, but I know I can do it on the system I have access to. Kasper
Best regards Daniel -------- Original-Nachricht -------- Betreff: Re: [R-sig-hpc] Rsge combined with mclapply? Von: Kasper Daniel Hansen <kasperdanielhansen at gmail.com> An: Daniel Saban?s Bov? <daniel.sabanesbove at ifspm.uzh.ch> Datum: Mit 13 Jun 2012 23:10:23 CEST
I am not too familiar with Rsge. ?But on our SGE cluster, the sysadmin has a specific type of parallel environment where you can do ? qsub -pe local 8-12 which asks for 8-12 cores on the same node. ?My guess is that once such a parallel resource has been set up, it should be trivial to use with Rsge. Kasper On Tue, Jun 5, 2012 at 6:19 AM, Daniel Saban?s Bov? <daniel.sabanesbove at ifspm.uzh.ch> ?wrote:
Dear Dan Bode, first of all I would like to thank you very much for your R-package "Rsge". It looks very helpful, because I need to submit my parallel jobs on the local cluster using SGE. Actually I have a trivially parallel problem which could be sequentially solved by lapply(), and on a single machine with multiple cores by mclapply() from the base R-package "parallel". Because the local cluster has nodes with multiple cores, I would like to combine sge.parLapply (to distribute to nodes) with mclapply (not lapply, to distribute to cores in each node). Before doing some modifications to sge.parParApply like introducing another apply.method for mclapply, I would like to be sure that you have not already programmed a nice solution for this problem. So, did you already have the same problem as I am facing now? Would you think another apply.method for sge.parParApply would be a good way to go? Thank you very much in advance for any hints or suggestions. Best regards Daniel Sabanes -- Daniel Saban?s Bov? University of Zurich Institute of Social and Preventive Medicine Division of Biostatistics Hirschengraben 84 CH-8001 Zurich Tel. +41 (0)44 634 49 73 Fax: +41 (0)44 634 43 86 URL : http://www.biostat.uzh.ch/aboutus/people/sabanes.html
_______________________________________________ R-sig-hpc mailing list R-sig-hpc at r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-hpc