Hello together! I've got a question regarding the assessment of stability of trading models. Here's a short intro to the background: I have fitted regression models with lagged independent variables to data of daily stock returns. (The model methodology was MARS, allowing for second order interaction terms.) The independent variables had been selected on basis of explorative clustering and correlation studies and showed face validity. I used both logit and gaussian link functions and generated out-of-sample predictions for a test window of two months (40+ observations, following the training window in time), which were of reasonable to really satisfying quality, depending on the exact model. In order to assess the stability of out-of-sample fit of a given model, I would normally draw cross-validation samples and partition them into training- and test subsets. Grounds for this would be the assumption of independent observations contained in the model and forced onto the data by backshifting them. However, I'm reluctant to believe that the data-generating process doesn't change over time, which is implied by my procedures. If this was true and time was not an issue, it should not be necessary to recalibrate the model, even after a long period of out-of-sample prediction. This seems overly optimistic to me. Returning to the question of stability assessment and cross-validation, I would like to know if there is some pragmatic solution. Is simple cross-validation viable? Do I need to go far into the past using some possibly sliding training- and test-windows? Or has anybody a different suggestion how to deal with this problem in the realm of regression models? Kind regards, Gero
Stability of trading models
2 messages · Gero Schwenk, Brian G. Peterson
Gero Schwenk wrote:
<... snip ...> In order to assess the stability of out-of-sample fit of a given model, I would normally draw cross-validation samples and partition them into training- and test subsets. Grounds for this would be the assumption of independent observations contained in the model and forced onto the data by backshifting them. However, I'm reluctant to believe that the data-generating process doesn't change over time, which is implied by my procedures. If this was true and time was not an issue, it should not be necessary to recalibrate the model, even after a long period of out-of-sample prediction. This seems overly optimistic to me. Returning to the question of stability assessment and cross-validation, I would like to know if there is some pragmatic solution. Is simple cross-validation viable? Do I need to go far into the past using some possibly sliding training- and test-windows? Or has anybody a different suggestion how to deal with this problem in the realm of regression models?
While we would all like a perfectly stable model, the reality is usually different. Since you've said that your model is regression-based, take a look at chart.RollingRegression. This will let you see the stability of your regression model over different rolling windows. I suggest checking longer, shorter, and from-inception windows.
From there, you'll have a lot of information to refine your modeling/tuning approach.
I generally am dubious that financial time series or trading models are completely stable over long periods of time.
Cheers,
- Brian
Brian G. Peterson http://braverock.com/brian/ Ph: 773-459-4973 IM: bgpbraverock