R, Nomad, HTCondor, etc... and future
David, Slurm is a good tool, especially if you are not doing complicated scheduling things with it. It is really designed to do HPC, so you might want to take a quick look at your needs and see whether HPC is really thing thing you want or whether you might be better off in an HTC environment, like HTCondor. They are really designed to do different things in different ways. Many, if not most, sites seem to end up building HPC clusters, but many of the users might be better off with HTC, instead. I'd counsel you to take a scan through the HTCondor documentation, and at Open Science Grid, just to get a sense of what the differences are. For example, with HTCondor, you could configure workstations to be part of your available resource pool during off hours, or if they are idle, and it's much harder to do that with something like Slurm. Anyway, you're buying the shoe, I would just make sure it fits well before walking a long way with it. -- bennet
On Thu, May 21, 2020 at 10:45 PM David Bellot <david.bellot at gmail.com> wrote:
I don't see in the above how your 'one-shot job' is different from your colleagues need to send spot requests.
You're right I didn't explain correclty. On one hand, I have experiments to run. Think about 'foreach %dopar%' loops and things like that. When it's done, I look at the result, and the work is done. My program has run and I don't it need anymore. On the other hand, they have many small services they want to keep waiting 24/7 and run when called, I mean on-demand. They don't need to be heavy on CPU, except for the few seconds, maybe when the services are called. In my use case, I don't need a service to stay up 24/7, but I use the CPU very intensively. And describing it like this now, I simply realized that solving these two different problems with one single solution seems a bit ... huh... silly :-) I found slurm reasonable in the past, and it has only gotten more widely
used / available sense. It will provide you with access to the compute resource, will account for 'who does what' and can schedule / resource (which I never really needed, and sounds like you don't either). Plus it will give you easy view on what is currently up or down, available etc pp. The devil is as always in the details. I'd say experiment and a little and take it from there.
I'll give Slurm a try then. You're not the first one to say it's a good
tool.
Thanks Dirk.
[[alternative HTML version deleted]]
_______________________________________________ R-sig-hpc mailing list R-sig-hpc at r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-hpc