Hey Folks,
So I'm slowly working my way through this. I've decided to try to get
things going using multiple cores on the same computer first, work out
the bugs there then try to move this over to multiple computers. I'm
using foreach/%dopar% (see the example below). When testing this on
different computers it seems that the environment gets transferred in
an inconsistent way :). I think I've managed to fix this with respect
to packages using the .packages option. However I've got functions
that I wrote myself and I've also run the model compiled. What I've
found that seems to work (more or less consistently) is the following:
n = 100
cores=detectCores()
cl <- makeCluster(cores[1]-1)
registerDoParallel(cl)
xall_p <- foreach(i=1:n,.packages=c("deSolve")) %dopar% {
source('myfunc.r')
dyn.load(paste("ode_model", .Platform$dynlib.ext, sep = ""))
x = run_simulation(args)
x }
stopCluster(cl)
You can see that within the loop I'm sourcing myfunc.r and I'm loading
the dynamic C library as well. Now my question :).
Is this the most appropriate way to do this? It seems, and this may be
my lack of understanding of the foreach function, that I'm trying to
load things at every iteration. Is this correct?
Thanks
John
Thanks
John
On Mon, Mar 27, 2017 at 7:45 AM, Tim Keitt <tkeitt at utexas.edu> wrote:
http://www.keittlab.org/ On Sat, Mar 25, 2017 at 9:53 AM, Daniel Kaschek < daniel.kaschek at physik.uni-freiburg.de> wrote:
Dear John, I wrote a function for my dMod package to automatize the scp and ssh processes. You can have a look at the runbg() function in https://github.com/dkaschek/dMod/blob/development/R/remoteComputing.R The approach might not be the most elegant. Furthermore, it is written with Linux on both the origin and target machine in mind. The function is documented and you might just have a look at it. Perhaps you need to modify it to your needs. If you have questions, let me know. Best regards, Daniel On Mon, 2017-03-20 at 13:02 +0530, John Harrold wrote:
Howdy Folks, I'm looking at parallelizing some of my code. Mostly I'm thinking about spreading the individual simulations in a Monte Carlo across a bunch of
That sounds like a task for one of the many parallel apply cluster functions in R. Create a vector of rng seeds and run one parallel job per seed? I would start with the parallel package. THK
computers. I'd like to spread this out across computers in a more
automated fashion as opposed to just adding cores or ssh-ing to
computers
is a somewhat manual fashion. I was going to get a free account on
AWS to
play around with, so I started looking through different tutorials
online.
There seems to be quite a few options, and I thought it was likely
someone
on this forum has some direct experience with this. And I'd like to
hear
about what folks are using in terms of packages, instances they are
using
on AWS (or other services if thats the case), etc.
Thanks
John
[[alternative HTML version deleted]]
_______________________________________________ R-sig-dynamic-models mailing list R-sig-dynamic-models at r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-dynamic-models
_______________________________________________ R-sig-dynamic-models mailing list R-sig-dynamic-models at r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-dynamic-models
[[alternative HTML version deleted]]
_______________________________________________ R-sig-dynamic-models mailing list R-sig-dynamic-models at r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-dynamic-models
------------------------------------- John M. Harrold john.m.harrold _at_gmail