doRedis on ec2:no new jobs being assigned to workers after first job completes
Nothing against doRedis, but for ec2 you could try something else... http://www.rinfinance.com/agenda/2012/workshop/WhitArmstrong.pdf -Whit
On Thu, Aug 23, 2012 at 9:00 AM, OpenTrades <jan at opentrades.nl> wrote:
I have been trying to use the R/Quantstrat applyParameters() function to analyze the performance for 1200 different Moving Average parameter combinations in the "Luxor" trading strategy. As I want to test with 6 years of GBP/USD pricing info at a 30 minute interval, each individual tests requires 15 minutes or more, so this is where doRedis comes into play. The platform that I am using is Amazon EC2. I configure one Redis server as a master, and from this master I also run the R program. Then I start up a bunch of slaves, containing 8 CPUs and 7GB of memory each (c1.xlarge in Amazon EC2 parlor), which automatically connect to the redis server, so I end up with a range of workers, which is a multiple of 8. Now if I test with only 10 days of data (instead of 6 years), or even 3 months, everything is great, whether I use 1 slave or 40 slaves (= 320 workers). But whenever I run with more than 1 year of data, I start getting problems, where workers will no longer be assigned a new job after the first one completes, and the worker count decreases in the redis-server window. Whenever this occurs, then if I restart the workers manually, they will log on to the redis server again and get assigned a new job; however once again no new job will be assigned to them after completion. It seems that this has something to do with the amount of time that it takes to complete the task. When renting multiple EC2 machines of the same instance type, they often appear to be slightly different nevertheless. And I found that, when running with 15 months of price data, one particular type (CPU freq = 2.13GHz / cache size=4096MB) gets assigned one job for each of its 8 workers, but then none to any worker anymore, where another much faster type (CPU freq=2.33GHz / cache size=6144MB) runs continuously without a flaw. FYI: all slaves share the same identical disk image. Anyone any ideas how to proceed and fix this? Or test I can do to find out what goes wrong? -- Jan Humme - OpenTrades WWW: http://www.opentrades.nl Email: jan at opentrades.nl Twitter: @opentrades
_______________________________________________ R-sig-hpc mailing list R-sig-hpc at r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-hpc