Message-ID: <3458b75c-fbb7-f9c1-b05a-f06c399f4443@idiv.de>
Date: 2016-11-24T07:30:52Z
From: Christian Krause
Subject: [parallel-package] feature request: set default cluster type via environment variable
Dear all,
I?m working as an administrator of a High-Performance Computing (HPC) Cluster which runs on Linux. A lot of people are using R on this Linux cluster and, of course, the *parallel* package to speed up their computations.
It has been our collective experience, that using |makeForkCluster| yields an overall better experience /on Linux/ than the |makePSOCKcluster|, for whatever definition of better. Let me just summarize that it works smoother. I believe, other people working with *parallel* on Linux can share this experience
Also, we did really welcome the environment variable |MC_CORES|, to be able to specify (in job submit scripts) the amount of CPU cores a user has been granted, most importantly for /dynamic resource requests/ (see below for an example).
What we would also appreciate - and now we finally get to the feature request - is another environment variable to choose the used cluster, as in:
|export MC_CLUSTER_TYPE=FORK |
Do you think something like this could be implemented in future releases?
Parallel R job submit script
This works with the Univa Grid Engine and should work with other * Grid Engine products:
|#!/bin/bash # request a "parallel environment" with 2 to 20 cores #$ -pe smp 2-20 # set number of cores for the R cluster to the granted value (between 2 and 20) export MC_CORES=$NSLOTS # we want this: export MC_CLUSTER_TYPE=FORK Rscript /path/to/script.R |
Best Regards
?
--
Christian Krause
Scientific Computing Administration and Support
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Phone: +49 341 97 33144
Email: christian.krause at idiv.de
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
German Centre for Integrative Biodiversity Research (iDiv) Halle-Jena-Leipzig
Deutscher Platz 5e
04103 Leipzig
Germany
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
iDiv is a research centre of the DFG ? Deutsche Forschungsgemeinschaft
iDiv ist eine zentrale Einrichtung der Universit?t Leipzig im Sinne des ? 92 Abs. 1 S?chsHSFG und wird zusammen mit der Martin-Luther-Universit?t Halle-Wittenberg und der Friedrich-Schiller-Universit?t Jena betrieben sowie in Kooperation mit dem Helmholtz-Zentrum f?r Umweltforschung GmbH ? UFZ. Beteiligte Kooperationspartner sind die folgenden au?eruniversit?ren Forschungseinrichtungen: das Helmholtz-Zentrum f?r Umweltforschung GmbH - UFZ, das Max-Planck-Institut f?r Biogeochemie (MPI BGC), das Max-Planck-Institut f?r chemische ?kologie (MPI CE), das Max-Planck-Institut f?r evolution?re Anthropologie (MPI EVA), das Leibniz-Institut Deutsche Sammlung von Mikroorganismen und Zellkulturen (DSMZ), das Leibniz-Institut f?r Pflanzenbiochemie (IPB), das Leibniz-Institut f?r Pflanzengenetik und Kulturpflanzenforschung (IPK) und das Leibniz-Institut Senckenberg Museum f?r Naturkunde G?rlitz (SMNG). USt-IdNr. DE 141510383
[[alternative HTML version deleted]]