A Layered Validation Pipeline for Backtest Overfitting
Summary
The article describes three complementary methods for reducing the chance that a strategy’s backtest reflects historical noise: Validation-within-Validation (V-in-V), Combinatorially Purged Cross-Validation (CPCV), and Combinatorially Symmetric Cross-Validation (CSCV). It identifies repeated hypothesis testing, curve-fitting, and researcher decisions informed by validation results as separate sources of bias. V-in-V assigns distinct roles to an exploratory training set, a limited-candidate validation set, and a final test opened once after the strategy is committed.
It also recommends anchored walkforward evaluation, which retains the original training start as later validation and test windows advance, and argues that CPCV and CSCV address leakage and selection bias in different stages of research. Multiple test windows can reveal regime sensitivity, though overlapping training histories make their results partly dependent and unsuitable for naive pooling. The article frames the methods as a combined research discipline rather than a guarantee of future performance. Its partition proportions and examples are guidance, and the excerpt does not establish that any specific strategy passes these safeguards.
Key ideas
- Repeated searches on the same historical data increase the chance of selecting a lucky result.
- V-in-V separates broad exploration, candidate screening, and a one-time final evaluation.
- Anchored walkforward testing expands the training history while advancing validation and test windows.
- CPCV and CSCV are presented as tools for controlling temporal leakage and selection bias.
- Overlapping training histories make walkforward test outcomes partly dependent, so results should not be naively pooled.
Tags
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.