Skip to content

R on a Beowulf Cluster

4 messages · Michael D, Paul Johnson, Dirk Eddelbuettel +1 more

Paul Johnson
# 
"Beowulf" is a generic term, it does not mean in particular what kind
of cluster framework your school has. For example, we run "Rocks"
cluster linux. You might contact your folks, find out what they are
doing. Or, if you can log in, you will be able to tell by snooping
around a bit in /etc, or by running

$ uname -a

You need to find out what scheduling framework your administrators
have set up and which system is your "head node" where you can log in
and submit jobs to the queue.

For work with R, you need the ordinary stuff for compiling R programs
(as you would in any workstation) and for the communications between
systems, get  OpenMPI.  Rmpi compiles against that, so don't bother
trying to make R go parallel until you have a working OpenMPI.  It is
possible to install that in a user account, but usually a sysadmin
will put it somewhere and tell you where the lib dir is for compiling
apps.  it will supply an executable "mpirun" or such that is named in
your scripts.

We have a new HPC system here and I've been trying to build user
documentaiton from the ground up because most of the students I'm
trying to help are not programmers either.

I have some online "working examples" I'm cultivating.  I work on this
pretty hard for a few weeks and then I get so frustrated I do other
things for a month or two. Parallel programming is very frustrating,
compared with the ordinary kind.

This URL should be all on one line:

/http://pj.freefaculty.org/cgi-bin/mw/index.php?title=Main_Page#2._High_Performance_Computing_on_a_Linux_Cluster
On Mon, Dec 20, 2010 at 1:54 PM, Michael D <mike409 at gmail.com> wrote:

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Dirk Eddelbuettel
    

    

      
      #
    

  


  

      
On 20 December 2010 at 14:54, Michael D wrote:

        
| Greetings HPC SIG,
| 
| I recently found out my school has a HPC project where they use Beowulf
| clusters and I was really hoping to wield this computing power before I
| finish my last semester; mostly to be acquainted with the workflow but also
| to run some large simulations for the course I'll be in.
| Suggestions for any interesting topics I might try are welcome.
| 
| What I really need is some idea of what libraries I would need to have in
| order to run some R jobs on the system. It will take a bit of negotiation
| with the admins to get the packages I need so I'd like to have as complete a
| list as possible to start. I can get the details of the setup if that would
| be necessary.
| 
| Also, I've seen the list of relevant packages (
| http://cran.r-project.org/web/views/HighPerformanceComputing.html) but my
| knowledge of computing only goes so far (my knowledge of HPC even less so)
| and my fall-back method of ctrl-f "beowulf" turned up no results.

We wrote a survey paper a little while back on this topic which may be a
suitable intro or refresher for you. It is linked from the HPC Task View and
is otherwise pretty easy to google too.

Enjoy,  Dirk

 
| Thanks for the help,
| 
| Michael DePersio
| M.S. Candidate
| Georgetown University Department of Mathematics and Statistics
| 
| 	[[alternative HTML version deleted]]
| 
| _______________________________________________
| R-sig-hpc mailing list
| R-sig-hpc at r-project.org
| https://stat.ethz.ch/mailman/listinfo/r-sig-hpc

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Liaw, Andy
    

    

      
      #
    

  


  

      
Actually "beowulf" is not really all that generic, I think.  We happen to have a Scyld Beowulf cluster (running ClusterWare on top of CentOS).  Beowulf is very different from other types of cluster in that it has a single master node (in most setup) and there are no login access to slave nodes.  If the master node goes down, the entire cluster is down.  The advantages are that it is single system image (there's one process table for the entire cluster), and no OS/software installation is needed on the slaves.

We've used Rmpi on the cluster.  NWS can work, but need some setup. 

Best,
Andy

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

        
Notice:  This e-mail message, together with any attachme...{{dropped:11}}