Parallelizing function with subfunctions using snowfall
If the functions that jr.parsimest calls are all defined in a package, then just load that package on each your workers, and they should be available to jr.parsimest. Of course, that means that the package has to be installed on all of the worker machines. - Steve
On Thu, Aug 25, 2011 at 4:39 AM, Joakim Ringblom <joakim.ringblom at ki.se> wrote:
Hi
I have a function (jr.parsimest) which is rather time consuming and since I want to do use it several times with different input I tried to parallelize the process using snowfall. The code works fine when running in serial with sfInit(parallel=FALSE) but I have a problem getting it to work with sfInit(parallel=TRUE, cpus=8).
The problem is that the function jr.parsimest contains several subfunctions that in turn have several more subfunctions and so on and it seems that I have to export all of them with sfExport or sfExportall. This will be rather tricky and tedious at best, since I don?t really know some of the subfunctions with subfunctions since they come from a non-Cran package not written by me. ?I also get a feeling that this is not the way to do things but I am not able confirm it anywhere.
Is my approach fundamentally wrong? Is there a better way of doing things?
Parallelizing a time-consuming subfunction is of course an idea but I rather parallelize the "topfunction" if possible.
I don?t think my code is important for the questions but I post it here anyhow just in case. I run the program on windows 7 computer at the moment but I would like to run it on unix later on if that matters.
Any help would be greatly appreciated.
/
Thanks in advance
Joakim Ringblom
PhD-student
Karolinska Institutet
Stockholm, Sweden
joakim.ringblom at ki.se
paramvect<-c(1,0.52, 1.3,4)
paramsd<-c(1,0.52, 1.3,4)
paramsd<-c(0,0,0,0)
ressd<-c(0.01)
doser<-c(0,0.381,0.475,0.564)
gruppstorlek<-c(10,10,10,10)
n<-1
nrdiff<-4
cef<-0.05
opiter<-2
searchnr<-1
bestnr<-gruppstorlek
outtable<-matrix(0,opiter,27+2*length(doser))
colnames(outtable)<-c("Medel-BMD","Medel-BMDL","Medel-BMDU","Medel-BMDL/BMD","Medel-BMDtrue", "Medel-SquaredError", "Medel-APE", "BMD-Varians","BMDL-Varians","BMDU-Varians","BMDL/BMD-Varians","BMDtrue-Varians", "SquaredError-Varians", "APE-Varians", "Antal BMD-NAs", "Parameter","Parameter","Parameter","Parameter", "SD","SD","SD","SD", "Ressd", 1:length(doser), 1:length(doser),"n", "CES", "searchnr")
for (k in 1:opiter){
?iternr<-1
?grupplist<-list()
?grupplist[[iternr]]<-gruppstorlek
?for (i in 1:length(doser)){
?arbnr<-gruppstorlek
?if (arbnr[i]>=nrdiff){
? arbnr[i]<-arbnr[i]-nrdiff
? for (j in 1:length(doser)){
? ?tempnr<-arbnr
? ?if (!(i==j)){
? ? searchnr<-searchnr+1
? ? iternr<-iternr+1
? ? tempnr[j]=tempnr[j]+nrdiff
? ? grupplist[[iternr]]<-tempnr
? ?}
? }
?}
?}
?sfInit(parallel=TRUE, cpus=8)
?svar<-sfLapply(grupplist, jr.parsimest, paramvect=paramvect, paramsd=paramsd, ressd=ressd, doser=doser, n=n, CES=cef, outtable=outtable,searchnr=searchnr)
?sfStop()
?besteffect<-svar[[1]]
?for (i in 2:length(svar)){
?if (svar[[i]][7]<besteffect[7]){
? besteffect<-svar[[i]]
? ? }
?}
?bestnr<-c(besteffect[29],besteffect[30],besteffect[31],besteffect[32])
?if(all(gruppstorlek==bestnr & nrdiff>1)){nrdiff<-nrdiff/2}
?gruppstorlek<-bestnr
?outtable[k,]<-besteffect
?write.table(outtable, file="utskrift.txt")
}
return(bestnr)
_______________________________________________ R-sig-hpc mailing list R-sig-hpc at r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-hpc