Off-Sample Testing to Evaluate Trading Strategy Robustness
Summary
The document explains why a profitable historical backtest may fail in live markets, especially when a strategy is tuned and judged on the same limited sample. It recommends splitting chronological price history into an earlier training segment for parameter selection and a later test segment for evaluating those choices. If results deteriorate substantially out of sample, that can indicate overfitting or a change in market conditions. A commodity-futures example illustrates comparing parameter sets and performance across periods.
It also describes rolling, recursive tests that repeatedly train on earlier data and evaluate on a later window, as well as cross-validation that rotates which segment is held out. The article notes that ordinary cross-validation can be misleading for time series because market regimes change and training on later data to test earlier data reverses chronology. Overlapping indicator lookbacks can also create correlated observations, complicating statistical inference. These methods help assess stability but cannot prove future profitability; historical evidence remains limited and strategies need sound underlying rationale.
Key ideas
- Choose parameters on earlier data and assess them on a later, untouched segment.
- Weak test-set performance relative to training results can signal overfitting or shifting market conditions.
- Rolling evaluation repeats chronological training and testing across successive windows.
- Cross-validation can use data inefficiently or violate time order when applied carelessly to markets.
- Overlapping indicator windows create dependence that can undermine some statistical tests.
Tags
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.