Using Train, Test, and Validation Sets in Walk-Forward Analysis

Summary

The document asks how machine-learning model validation fits within walk-forward analysis or optimization. It contrasts choosing strategy parameters on a training period and measuring them on a test period with the machine-learning practice of reserving a validation set for hyperparameter selection. The answer recommends treating trading strategy backtests like model validation and keeping validation data aside from iterative strategy development as a final assessment period.

It emphasizes maximizing out-of-sample evaluation while cautioning that backtest performance alone is weak assurance of future success. A strategy also needs a plausible market inefficiency that explains why it should work and when that edge may decay. The answer argues that machine-learning approaches often fail when the supposed inefficiency is unclear, making performance deterioration difficult to anticipate. The discussion gives general guidance rather than a detailed walk-forward schedule, leakage controls, or empirical comparisons of split designs; it also uses “validation” for a final untouched crucible, which differs from the common ML use of validation data for model selection.

Key ideas

• Walk-forward strategy evaluation can use separate training, testing, and validation periods.
• Training data fits model parameters, while validation data can guide hyperparameter choices.
• An untouched out-of-sample period should be kept separate from iterative strategy development.
• Backtest results deserve caution because they may not persist in future market conditions.
• A strategy needs a clear underlying inefficiency to explain why its edge exists and how it may decay.

Tags

# Validation set on Walk Forward Analysis


# Validation set on Walk Forward Analysis












When backtesting a trading strategy using Walk Forward Analysis/Optimization, I see people split each window into training and testing sets.

Suppose you want to select the best combination of MAs for your MACD strategy, you would first try the different combinations on your training set, select the best one, and report the results you get with those parameters on your test set.

However, when the strategy in place is a machine-learning model, shouldn't we have a validation set as we have in every other ML problem? The training set would be for learning the model parameters, the validation set to select the best-performing hyper-parameters, and finally, the test set to report the model results.

I haven't seen anywhere where a validation set is used in WFA/WFO. I would appreciate any input or extra resources on this topic.

Cheers!

## Answer by DerivativesTamer (score 1)

https://quant.stackexchange.com/a/74273

You should treat backtesting strategies exactly as you would treat validating a machine learning model — look for ways to maximise your out-of-sample testing periods.

This means train-test-validation sets, with the validation being totally aside from your iteration and strategy development process, a sort of final crucible for your strategy.

As always, take the back test with a pinch of salt, it is much more important to reconcile your views of the forward path of markets with the underlying inefficiency the strategy is built to exploit - this is where most machine learning tools fail as the underlying inefficiency the model is apparently learning is somewhat unclear and thus it is unclear when the models performance will start to decay.

Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)

This summary was written by Stratmill's research agent from the original; it is not a copy of the source.