Estimating Backtest Overfitting Probability with CSCV

Summary

The document explains how combinatorially symmetric cross-validation (CSCV) can estimate the probability of backtest overfitting (PBO). It distinguishes training overfitting, where a model performs poorly on held-out data, from backtest overfitting, where a strategy’s historical performance fails to carry into live trading. CSCV assesses the latter by checking whether the strategy that ranks best in-sample also ranks well out-of-sample.

The method divides the historical period into equal segments, repeatedly combines half into a training set and uses the remainder as a test set, then ranks strategies by a performance measure such as Sharpe ratio. The share of splits where the in-sample winner ranks in the lower half out-of-sample is treated as PBO. Three reported examples show lower PBO for two multifactor stock-selection studies and higher PBO for a moving-average timing strategy. The document cautions that segment length affects both computation and ranking stability, and suggests choosing a metric suited to the portfolio, such as information ratio for index enhancement. These case results illustrate the method but do not establish that low PBO guarantees live success.

Key ideas

• CSCV repeatedly partitions backtest history into complementary training and test samples.
• PBO estimates how often the in-sample winner ranks in the lower half out-of-sample.
• The examples report lower PBO for two multifactor selection cases and higher PBO for a timing case.
• Shorter segments increase the number of combinations and computation, while longer segments can make rankings more sensitive to chance.
• The ranking metric should fit the strategy, and PBO does not remove market or live-trading risk.

Tags

This summary was written by Stratmill's research agent from the original; it is not a copy of the source.