[Bioc-devel] BiocParallel -- update

On Tue, Dec 4, 2012 at 12:47 PM, Ryan C. Thompson
<rct at thompsonclan.org <mailto:rct at thompsonclan.org>> wrote:

    One issue that I see is that for some kinds of parallel backends,
    there may not be any way for "bpworkers" to return something
    meaningful. For example, a backend that submits jobs to a large
    cluster may not know exactly how many nodes are in the cluster,
    and in any case returning the total number of nodes may not be
    appropriate, since those nodes are shared with other cluster
    users. This is primarily important for the pvec function, which
    uses the result of bpworkers to decide how many chunks to split
    the input into.

    I guess one solution is to make sure that for any backend that
    cannot natively determine a number of available workers, we
    require the number of workers as an argument when creating the
    param object for that backend. e.g.:

    param <- IndeterminateSizedClusterParam__(workers=50).


I think this is on the right track. Since the nature of the request
affects how the jobs are scheduled (earlier or later), there's no way
to automatically make the decision, even if we could detect the total
cluster size. As I noted in the previous email, having a consistent
means of specifying resource requests across backends would be helpful.

I could see an API like:

request <- ResourceRequest(num.cores = 5)
cluster <- LSFCluster(request) # or MulticoreCluster(request)
pvec(v, cluster = cluster)

Depending on the cluster, the 'cluster' object could be queried for
whether the requested resources are currently available (or the jobs
will need to wait). A default cluster object could be registered in
options(). The Cluster constructors could take the arguments of
ResourceRequest directly for simple tasks.

Then the question is whether pvec returns the result of evaluation, or
the promise of evaluation. Probably best to have pvec always behave
synchronously, then have variants like apvec() for asynchronous
execution. The promise would be backend-specific and support status
queries. For multicore, this is basically mcparallel/mccollect.

    Additionally, as discussed previously, it makes sense to be able
    to explicitly choose a chunk size or number of chunks for pvec,
    rather than splitting into exactly as many chunks as there are
    parallel workers. I implemented this in the non-generic
    multicore-only version of pvec, but I still need to port it to the
    generic version that works for any param. Do people think that the
    chunk options should be included in the MulticoreParam class, or
    specified when pvec is called?


What about supporting both? If passed directly to pvec, the params
option is overridden.

    I have also written a non-generic multicore-only version of
    pvectorize that allows for multiple vectorized arguments instead
    of just one, and furthermore gives the parallelized function an
    identical signature to the original function. Again, this needs to
    be ported to the generic bpvectorize.


Awesome.

Michael