Using CSCV to Estimate Backtest Overfitting Probability

Summary

This study explains combinatorially symmetric cross-validation (CSCV) as a way to estimate the probability that a strategy selected as best in a backtest will rank poorly across alternative partitions of that history. The backtest period is divided into blocks; combinations of half the blocks form training samples, with the remainder serving as test samples. The selected strategy’s test-period rank is tracked, and the share of cases in the bottom half is treated as the estimated probability of backtest overfitting. Performance is usually ranked by Sharpe ratio, though a metric such as information ratio may better fit an index-enhancement portfolio.

The report applies the method to machine-learning equity factor models, different cross-validation schemes, and a moving-average timing strategy. It reports lower estimated overfitting for the stock-selection comparisons and higher estimates for the timing model’s chosen parameter sets. These are historical case studies, not guarantees about live results. Block size affects both computation and ranking stability, and shuffling historical segments simplifies away path dependence; changing markets and noise fitting remain important limits.

Key ideas

• CSCV estimates backtest overfitting by checking how the in-sample winner ranks across complementary out-of-sample partitions.
• The estimated probability is the frequency with which the selected strategy falls in the bottom half of test-period rankings.
• Use a performance measure suited to the strategy, such as information ratio for benchmark-relative portfolios.
• Smaller time blocks create more partitions but require more computation, while larger blocks can make rankings less stable.
• The case studies found lower estimates for equity factor model comparisons than for the moving-average timing parameter choices.

Tags

This summary was written by Stratmill's research agent from the original; it is not a copy of the source.