Running jobs on a linux cluster: ways to communicate finished jobs, so fill in with new jobs

Dear all:

I would appreciate any help you are willing to offer. I have a simulation
program that runs serially. However, I would like to run the jobs in such a
way that when a simulation is finished another job can begin to run. ?The
simulations take different amounts of time, so it would be ideal to have a
way to communicate that jobs are done, and to initiate new jobs. The linux
cluster IT staff at my institution do not have much documentation or
experience with running R jobs. ?I am new to HPC, so my apologizes for this
potentially very basic inquiry.

Thank you for your time and consideration,
Laura

You don't give us much to go on. What scheduler does your cluster use?
for example.

Here's what I'd do. Write a shell script that runs all of the programs
one after the other.  Without knowing more about the scheduling scheme
on your cluster, I can't say exactly how I would go about it.

If you have access to a BASH shell, for example, it should be as simple as

#!/bin/bash

R --vanilla -f yourRprogram1.R

R --vanilla -f yourRprogram2.R

=====================

and so forth. If you rewrite the first line of your R code to use
Rscript or littler, then you don't even need to bother with the "R
--vanilla -f" part, as each R program will become self aware (and take
over the world, like in Terminator).

If you run exactly the same R program over and over again, make a for loop.

As long as you have the details worked out on each individual run of
the model, the rest of it is not even really a "cluster" problem. You
have to run one after the other.

FYI, I've been uploading practical working examples for our Rocks
Linux cluster using the Torque/OpenPBS scheduling system.  Maybe some
will help you.

http://pj.freefaculty.org/cgi-bin/mw/index.php?title=Cluster:Main

I think I could work out an example of the sort you describe if you
tell us a bit more about how the separate simulation runs talk to each
other.

Or, I should add, if the runs go one after the other, why don't you
put them all in 1 R program.  ??

Paul E. Johnson
Professor, Political Science
1541 Lilac Lane, Room 504
University of Kansas