dear developers, i'm adding parallel support to my package by copying & pasting the way in which this is done by other packages like ShortRead. i'm currently using 'snow' and since i'm providing this support for computing-intensive calculations i'd like that while these calculations are running in parallel, some sort of progress is reported. i'd like to ask whether anybody here has added parallel support to his/her package including some sort of progress reporting functionality. my first guess is that probably the loop of slave R processes created by 'snow' should be replaced by something else, but i hope somebody has a more clear idea about what could be done. thanks! robert.
[Bioc-devel] reporting progress while running in parallel
4 messages · Robert Castelo, Martin Morgan, Ralf Tautenhahn
On 08/27/2010 07:27 AM, Robert Castelo wrote:
dear developers, i'm adding parallel support to my package by copying & pasting the way in which this is done by other packages like ShortRead. i'm currently using 'snow' and since i'm providing this support for computing-intensive calculations i'd like that while these calculations are running in parallel, some sort of progress is reported. i'd like to ask whether anybody here has added parallel support to his/her package including some sort of progress reporting functionality. my first guess is that probably the loop of slave R processes created by 'snow' should be replaced by something else, but i hope somebody has a more clear idea about what could be done.
Hi Robert -- I don't have a solution, but if I were starting on this with snow I'd look at clusterApplyLB and dynamicClusterApply -- break the overall set of tasks into a number of jobs > the number of nodes, and then in a loop like that in dynamicClusterApply report to the user as each job comes back from recvOneResult. In Rmpi a fun project might be to figure out how to use a separate comm to send errors. Martin
thanks! robert.
_______________________________________________ Bioc-devel at stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel
Martin Morgan Computational Biology / Fred Hutchinson Cancer Research Center 1100 Fairview Ave. N. PO Box 19024 Seattle, WA 98109 Location: Arnold Building M1 B861 Phone: (206) 667-2793
XCMS has slightly modified versions of snow's clusterApply(LB) that allow progress reporting. Regards, Ralf. schrieb Robert Castelo:
dear developers, i'm adding parallel support to my package by copying& pasting the way in which this is done by other packages like ShortRead. i'm currently using 'snow' and since i'm providing this support for computing-intensive calculations i'd like that while these calculations are running in parallel, some sort of progress is reported. i'd like to ask whether anybody here has added parallel support to his/her package including some sort of progress reporting functionality. my first guess is that probably the loop of slave R processes created by 'snow' should be replaced by something else, but i hope somebody has a more clear idea about what could be done. thanks! robert.
_______________________________________________ Bioc-devel at stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel
2 days later
Ralf, thanks for the hint, i've looked at the current devel code and over the original snow code of staticClusterApply() it just adds a message to inform that a job is being sent to a slave. however, as i was saying in a previous message to Martin, i'm not interested in reporting when a job starts and ends but how far are the slaves in their respective jobs. again, if you or anybody has ever seen something like this in an R or BioC package, i'll be very happy to hear about it. cheers, robert.
On Sat, 2010-08-28 at 03:41 -0700, Ralf Tautenhahn wrote:
XCMS has slightly modified versions of snow's clusterApply(LB) that allow progress reporting. Regards, Ralf. schrieb Robert Castelo:
dear developers, i'm adding parallel support to my package by copying& pasting the way in which this is done by other packages like ShortRead. i'm currently using 'snow' and since i'm providing this support for computing-intensive calculations i'd like that while these calculations are running in parallel, some sort of progress is reported. i'd like to ask whether anybody here has added parallel support to his/her package including some sort of progress reporting functionality. my first guess is that probably the loop of slave R processes created by 'snow' should be replaced by something else, but i hope somebody has a more clear idea about what could be done. thanks! robert.
_______________________________________________ Bioc-devel at stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel
_______________________________________________ Bioc-devel at stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel