Using CSCV to Estimate Backtest Overfitting in Strategy Optimization

Summary

The article explains Combinatorially Symmetric Cross Validation (CSCV) as a way to estimate the Probability of Backtest Overfitting (PBO) after trying multiple strategy parameter sets. It starts from consistent, granular performance data for every trial, such as bar-level returns or profit and loss. The data is divided into an even number of blocks; combinations of half the blocks form in-sample sets, while the remainder form corresponding out-of-sample sets. For each split, the best in-sample configuration is ranked against the other configurations out of sample, and the accumulated relative ranks are used to estimate PBO.

The article describes an MQL5 implementation that accepts a performance criterion and whether it should be maximized or minimized, then demonstrates using the method with an Expert Advisor. It emphasizes that the result depends on the trial set and optimization period: too few configurations may understate overfitting, while unrealistic trials may inflate it. CSCV also requires access to comparable per-trial data, which may be difficult for systems whose source or outputs are unavailable. It is a diagnostic estimate, not a guarantee of future performance.

Key ideas

• CSCV estimates how often an in-sample winner ranks poorly out of sample across combinatorial data splits.
• The input matrix needs consistent granular performance observations for every parameter trial.
• The performance metric can vary, provided the method receives the corresponding underlying data.
• PBO estimates depend on the number and realism of tested configurations and on the chosen period.
• CSCV is a diagnostic for optimization overfitting, not proof that a strategy will generalize.

Tags

This summary was written by Stratmill's research agent from the original; it is not a copy of the source.