On Thu, Jun 14, 2012 at 4:42 AM, Daniel Saban?s Bov?
<daniel.sabanesbove at ifspm.uzh.ch> wrote:
Hi Kasper,
thank you for this hint! I was not aware of this possibility. Yes, indeed,
then it should be trivial to let multiple jobs run on cores of the same
node. In the meantime I have found a solution by tweaking the Rsge package
to use mclapply() from the base R-package "parallel". This also works.
This might work in the sense that it produces workable code on your
system. But are you sure that you do not over-use your resources.
For example, on our system, I can qsub a job that SGE thinks takes up
a single slot (= core, the default) and then still use mclapply over 8
cores. In that situation, I have asked for just 1 core, but I am
really using 8. This has clear implications for the other users on
the system as well as resource allocation. This is because our SGE
system is not too tightly locked down - it may be possible for a
sysadmin to prevent this, but I know I can do it on the system I have
access to.
Kasper
Best regards
Daniel
-------- Original-Nachricht --------
Betreff: Re: [R-sig-hpc] Rsge combined with mclapply?
Von: Kasper Daniel Hansen<kasperdanielhansen at gmail.com>
An: Daniel Saban?s Bov?<daniel.sabanesbove at ifspm.uzh.ch>
Datum: Mit 13 Jun 2012 23:10:23 CEST
I am not too familiar with Rsge. But on our SGE cluster, the sysadmin
has a specific type of parallel environment where you can do
qsub -pe local 8-12
which asks for 8-12 cores on the same node. My guess is that once
such a parallel resource has been set up, it should be trivial to use
with Rsge.
Kasper
On Tue, Jun 5, 2012 at 6:19 AM, Daniel Saban?s Bov?
<daniel.sabanesbove at ifspm.uzh.ch> wrote:
Dear Dan Bode,
first of all I would like to thank you very much for your R-package
"Rsge".
It looks very helpful, because I need to submit my parallel jobs on the
local cluster using SGE.
Actually I have a trivially parallel problem which could be sequentially
solved by lapply(), and on a single machine with multiple cores by
mclapply() from the base R-package "parallel". Because the local cluster
has
nodes with multiple cores, I would like to combine sge.parLapply (to
distribute to nodes) with mclapply (not lapply, to distribute to cores in
each node).
Before doing some modifications to sge.parParApply like introducing
another
apply.method for mclapply, I would like to be sure that you have not
already
programmed a nice solution for this problem. So, did you already have the
same problem as I am facing now? Would you think another apply.method for
sge.parParApply would be a good way to go?
Thank you very much in advance for any hints or suggestions.
Best regards
Daniel Sabanes
--
Daniel Saban?s Bov?
University of Zurich
Institute of Social and Preventive Medicine
Division of Biostatistics
Hirschengraben 84
CH-8001 Zurich
Tel. +41 (0)44 634 49 73
Fax: +41 (0)44 634 43 86
URL : http://www.biostat.uzh.ch/aboutus/people/sabanes.html