Parallelizing applyStrategy to multiple symbols

I suspect you're running up against communication and memory management
time and resource contention. ?

applyIndicators and applySignals should all be using vectorized code,
so the potential benefit from parallelization will likely be negative,
as communication and memory management swap any benefit from the
calculations.

applyRules might benefit from parllelization, but you would need to
come back together on any rebalancing period.  You would also have
significant copying time.

If you were going to make this work, you'd need to minimize copies. 
Your effective 'reduce' operation at the end by only returning
tradeStats could do this for the end of the calculation, but at the
start, you'd need to be smarter about how you segment market data to
each worker. 

Just putting getSymbols on the workers might run into I/O contention
issues.  You also don't need to redeclare the strategy object.  You
could just copy that to each worker.

When we've done things as a one-off, we typically create portfolios for
each segment, and try to avoid as many copies as we can.

You'd need to profile to see exactly where you're getting hung up, but
this approach seems too simplistic (see my first sentence for hints).

We haven't bothered to do this in the package itself since with a
little work we can usually get to around one core minute per symbol per
day on L1 tick data, which means that even a large backtest on tick
data can finish in a few hours.  The cost of optimizing execution time
doesn't seem to be worth the cost in programming and testing time.

Regards,

Brian

Parallelizing applyStrategy to multiple symbols

Thread (5 messages)