Parallelizing function with subfunctions using snowfall
Hi Stephen: Thank you for answering, I really appreciate it. However, the workers in this case are the different cores on my octa-core processor and I have the package installed and loaded on my system. Yes, I do load the package in every session. I can?t install/load the package on to a core, can I? Any ideas from anyone are appreciated. By the way, the error message I get while running is: Error in checkForRemoteErrors(val) : 8 nodes produced errors; first error: could not find function "f.ini" ? F.ini is a package-function within jr.parsimest. Thanks in advance. / Joakim Ringblom PhD-student Karolinska Institutet, Institutet f?r Milj?medicin Nobels v?g, 13 Box 210 17177 Stockholm Sweden +46- (0)733965553 ________________________________________ Fr?n: Stephen Weston [stephen.b.weston at gmail.com] Skickat: den 25 augusti 2011 15:22 Till: Joakim Ringblom Kopia: r-sig-hpc at r-project.org ?mne: Re: [R-sig-hpc] Parallelizing function with subfunctions using snowfall If the functions that jr.parsimest calls are all defined in a package, then just load that package on each your workers, and they should be available to jr.parsimest. Of course, that means that the package has to be installed on all of the worker machines. - Steve
On Thu, Aug 25, 2011 at 4:39 AM, Joakim Ringblom <joakim.ringblom at ki.se> wrote:
Hi
I have a function (jr.parsimest) which is rather time consuming and since I want to do use it several times with different input I tried to parallelize the process using snowfall. The code works fine when running in serial with sfInit(parallel=FALSE) but I have a problem getting it to work with sfInit(parallel=TRUE, cpus=8).
The problem is that the function jr.parsimest contains several subfunctions that in turn have several more subfunctions and so on and it seems that I have to export all of them with sfExport or sfExportall. This will be rather tricky and tedious at best, since I don?t really know some of the subfunctions with subfunctions since they come from a non-Cran package not written by me. I also get a feeling that this is not the way to do things but I am not able confirm it anywhere.
Is my approach fundamentally wrong? Is there a better way of doing things?
Parallelizing a time-consuming subfunction is of course an idea but I rather parallelize the "topfunction" if possible.
I don?t think my code is important for the questions but I post it here anyhow just in case. I run the program on windows 7 computer at the moment but I would like to run it on unix later on if that matters.
Any help would be greatly appreciated.
/
Thanks in advance
Joakim Ringblom
PhD-student
Karolinska Institutet
Stockholm, Sweden
joakim.ringblom at ki.se
paramvect<-c(1,0.52, 1.3,4)
paramsd<-c(1,0.52, 1.3,4)
paramsd<-c(0,0,0,0)
ressd<-c(0.01)
doser<-c(0,0.381,0.475,0.564)
gruppstorlek<-c(10,10,10,10)
n<-1
nrdiff<-4
cef<-0.05
opiter<-2
searchnr<-1
bestnr<-gruppstorlek
outtable<-matrix(0,opiter,27+2*length(doser))
colnames(outtable)<-c("Medel-BMD","Medel-BMDL","Medel-BMDU","Medel-BMDL/BMD","Medel-BMDtrue", "Medel-SquaredError", "Medel-APE", "BMD-Varians","BMDL-Varians","BMDU-Varians","BMDL/BMD-Varians","BMDtrue-Varians", "SquaredError-Varians", "APE-Varians", "Antal BMD-NAs", "Parameter","Parameter","Parameter","Parameter", "SD","SD","SD","SD", "Ressd", 1:length(doser), 1:length(doser),"n", "CES", "searchnr")
for (k in 1:opiter){
iternr<-1
grupplist<-list()
grupplist[[iternr]]<-gruppstorlek
for (i in 1:length(doser)){
arbnr<-gruppstorlek
if (arbnr[i]>=nrdiff){
arbnr[i]<-arbnr[i]-nrdiff
for (j in 1:length(doser)){
tempnr<-arbnr
if (!(i==j)){
searchnr<-searchnr+1
iternr<-iternr+1
tempnr[j]=tempnr[j]+nrdiff
grupplist[[iternr]]<-tempnr
}
}
}
}
sfInit(parallel=TRUE, cpus=8)
svar<-sfLapply(grupplist, jr.parsimest, paramvect=paramvect, paramsd=paramsd, ressd=ressd, doser=doser, n=n, CES=cef, outtable=outtable,searchnr=searchnr)
sfStop()
besteffect<-svar[[1]]
for (i in 2:length(svar)){
if (svar[[i]][7]<besteffect[7]){
besteffect<-svar[[i]]
}
}
bestnr<-c(besteffect[29],besteffect[30],besteffect[31],besteffect[32])
if(all(gruppstorlek==bestnr & nrdiff>1)){nrdiff<-nrdiff/2}
gruppstorlek<-bestnr
outtable[k,]<-besteffect
write.table(outtable, file="utskrift.txt")
}
return(bestnr)
_______________________________________________ R-sig-hpc mailing list R-sig-hpc at r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-hpc