R, Nomad, HTCondor, etc... and future

Hi David,

I live in a large HPC world, where distributed computing is inherently batch, so take my advice with that perspective. Large systems are mostly incompatible with the interactive concept of "backend" and instead support SPMD-style batch programming. SPMD is mostly MPI+X, meaning that the distributed aspect is handled by MPI and within node aspects can vary among several options including MPI, fork, OpenMP, and OpenACC. But even on a medium slurm-managed cluster (possibly in a corporate environment), for R I would recommend a combination of pbdR.org distributed packages and parallel package's mclapply components for within node parallelism.

Best,
George

?-----Original Message-----
From: R-sig-hpc <r-sig-hpc-bounces at r-project.org> on behalf of David Bellot <david.bellot at gmail.com>
Date: Thursday, May 21, 2020 at 8:26 PM
To: <r-sig-hpc at r-project.org>
Subject: [R-sig-hpc] R, Nomad, HTCondor, etc... and future

    Hi R HPC,

    I was wondering if anyone has ever used Nomad from Hashicorp as a backend
    engine to run R distributed code.
    Moreover, if you use it with *future, *I'd love to hear about your
    experience and if you published a package for it, I'd love to use it too.

    If not, which other distribution engine do you use? (apart from those
    supported in *batchtools*)?
    HTCondor, DockerSwarm, BOINC, etc... ?

    I didn't make a final decision yet on which engine to use, but it has to be
    versatile enough (I know it's not a lof of information but think about
    "corporate environment" with various needs. My R need is just one among
    many other).

    Thanks for your help.
    David


    _______________________________________________
    R-sig-hpc mailing list
    R-sig-hpc at r-project.org
    https://stat.ethz.ch/mailman/listinfo/r-sig-hpc