Nested Walk-Forward Cross-Validation for Financial Machine Learning

Summary

The article presents a temporal nested cross-validation design intended to reduce optimistic evaluation in financial machine learning. It divides observations into an outer training zone, an inner validation checkpoint, and a final test segment that is opened once. Within the training zone, an inner purged walk-forward search selects hyperparameters using the one-standard-error rule, which favors a simpler configuration near the best score. An outer loop then evaluates those choices on folds not used in selection, using either walk-forward splits or combinatorial purged cross-validation.

For probability calibration, it generates out-of-fold predictions so the calibrator does not learn from in-sample outputs, and aggregates fold results into consensus parameters and a final calibration map. The article explains leakage risks from overlapping labels, temporal dependence, regime changes, repeated tuning, and reopening a test set. It provides implementation details and references, but the design cannot eliminate nonstationarity or guarantee unbiased performance under every practical choice. The final test is meaningful only if it remains untouched after evaluation.

Key ideas

• Use chronological data zones to separate model selection, a validation checkpoint, and the final evaluation.
• Purging and embargo help limit leakage when label horizons overlap across temporal folds.
• The one-standard-error rule selects a simpler configuration when its score is statistically close to the best.
• Generate out-of-fold predictions for calibration so the calibrator does not rely on training-set predictions.
• Open the final test once and do not tune the model after inspecting its result.

Tags

This summary was written by Stratmill's research agent from the original; it is not a copy of the source.