Why Cross-Validation Can Still Overfit Backtests

Summary

The document examines whether cross-validation prevents backtest overfitting. It contrasts a common concern about repeatedly applying holdout tests with the claim that results should instead report the average out-of-sample performance across all trials. The key issue is whether every trial is disclosed and included in the reported analysis.

The response says cross-validation can be informative when researchers publish the full set of trial outcomes. In typical practice, however, researchers may report only the selected successful result. Repeated testing followed by selective disclosure creates a multiple-testing problem: the reported result can appear more statistically convincing than the testing process warrants. The document offers no empirical study or detailed procedure for correcting this bias, and its discussion is brief. Its main lesson is that cross-validation alone does not guarantee protection from overfitting when trial history is hidden; transparent reporting of all trials matters.

Key ideas

• Repeated holdout testing increases the chance that an invalid strategy appears successful.
• Reporting only a selected trial can conceal the effects of multiple testing.
• Publishing results across all trials makes the testing process more transparent.
• Cross-validation does not by itself eliminate backtest overfitting when trial outcomes are selectively reported.

Tags

# why does Cross Validation *not* solve Backtest overfitting?


# why does Cross Validation *not* solve Backtest overfitting?












In this famous paper, Bailey and De Prado discard Cross Validation as tool to check for Backtest overfitting, on the ground that it is just an holdout method:

> ... If we apply the holdout method enough times (say 20 times for a 95% confidence level), false positives are no longer unlikely: They are expected. The more times we apply holdout, the more likely an invalid strategy will pass the test, which will then be published as a single-trial outcome ...

But publishing the results as a single-trial outcome is a misuse of Cross Validation. One should publish the average OOS performance of the K trials. So Bailey and De Prado don't have a point there. Cross Validation does solve the problem of backtest overfitting.

Am I missing something?

## Answer by Jase (score 8)

https://quant.stackexchange.com/a/65544

If they publish information about all K trials, then you're right. But the author's point is that that's not typical practice. Typical practice is to not disclose that information, and it amounts to p-hacking where the statistical power of the test differs to what's being advertised.

Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)

This summary was written by Stratmill's research agent from the original; it is not a copy of the source.