Using Bootstrap Tests and Cross-Validation to Assess Strategy Overfitting

Summary

The document asks whether bootstrap resampling should happen before strategy data mining or after a candidate strategy has been selected. It describes a monthly SPY and TLT rebalancing strategy tested over six years and 72 trades, reporting a backtest Sharpe ratio of 2.06 that falls to 1.04 on bootstrapped data. The author wonders whether optimization should restart with the bootstrapped results as the target.

The response says resampling can provide a more cautious view of performance, while warning that a bootstrap may fail to preserve financial time series structure, including changing autocorrelation and relationships across assets. It also points to cross-validation as a way to reduce overfitting. The exchange does not specify a bootstrap design, resampling unit, or validation protocol, and the small trade count makes the reported performance especially context-dependent. It offers methodological cautions rather than evidence that one ordering will produce a reliable strategy.

Key ideas

• Bootstrap results can give a more conservative view of a strategy’s measured performance.
• Simple resampling may not preserve time dependence or cross-asset relationships.
• Cross-validation can help reduce overfitting during strategy selection.
• The reported Sharpe decline is specific to the described strategy and bootstrap setup.
• The document does not prescribe a precise resampling or validation procedure.

Tags

# Bootstrapping first, then data mine?


# Bootstrapping first, then data mine?












I posted this on Cross Validated a month ago, but I didn't get any answers. Sorry for the second post!

I'm wondering about the whole process of testing/data-mining for a strategy and THEN testing on bootstrapped data. Does it make sense to bootstrap your data first and then data-mine for the best result?

This came to me because my current strategy had significant decay when I tested it against the bootstrapped data. The initial strategy is one that re-balances between the SPY and the TLT on a monthly basis. Back-tested on 6 years worth of data (or 72 total trades), the strategy has a Sharpe of 2.06. But when I test it against the bootstrapped data, the Sharpe ratio drops significantly to 1.04.

It seems like I'm ultimately looking to raise the 1.04 number, and to do that I'll need to

- start the data-mining process over again to find a new back-tested strategy

- test again against the bootstrapped data.

So can I skip step #1? Obviously this will require some significant computational power.

## Answer by pat (score 2)

https://quant.stackexchange.com/a/4650

It's good in the sense that the bootstrap will give you a more sobering view of reality. However, it is fairly difficult to perfectly capture all the relationships of financial assets via bootstrapping (dynamic auto and cross correlations, etc..). It would be great to see some comments on what people have been using. From a data mining perspective, you might want to look into cross validation techniques as it helps you to avoid over-fitting.

Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)

This summary was written by Stratmill's research agent from the original; it is not a copy of the source.