Why Trading Strategies Lose Effectiveness: Overfitting and Decay
Summary
The document describes two reasons a trading strategy may perform worse after research or deployment: the original evidence overstated an effect, or a real effect weakened over time. It explains how analytical choices and multiple opportunities to discover patterns can produce results that look statistically convincing but fail to reproduce on the same data or replicate on new data. A failure to account for cross-sectional correlation is given as an example of how estimated standard errors can be badly distorted.
The second explanation is that a strategy based on investor behavior may lose its edge once that behavior becomes widely understood and traders adapt. Tax-loss selling and subsequent repurchasing illustrate this possibility. The discussion cautions that these explanations can be difficult to distinguish in finance, where researchers often rely on overlapping historical datasets and must wait for new observations. It offers conceptual reasoning and references, rather than a test for diagnosing a particular strategy or evidence quantifying how quickly an effect decays.
Key ideas
- A strategy can disappoint because its backtest overstated an effect or because a genuine effect later weakened.
- Many research choices create opportunities to find patterns that will not replicate.
- Ignoring cross-sectional dependence can distort estimates of statistical uncertainty.
- Behavioral effects may shrink as market participants learn about and respond to them.
- In finance, limited access to wholly new data makes overestimation and decay difficult to separate.
Tags
Full text
# Why do trading strategies lose effectiveness over time? # Why do trading strategies lose effectiveness over time? I don't do any financial trading, and this question came up to me as I was browsing through some questions on this SE. A general sentiment I felt after reading though several questions on this SE, is that algorithmic trading strategies generally won't remain as effective as they were during backtesting, or after some time in deployment. On some intuitive level I can understand why that might be the case, it just seems odd. But I was wondering if there is some in-depth reasoning on why this is the case. ## Answer by Matthew Gunn (score 15, accepted) https://quant.stackexchange.com/a/35822 There are two key concerns (which in practice, may be difficult to distinguish): - Previous research overestimated an effect. - The effect shrinks over time. #### 1. Problems with reproducibility and replicability Previous research may have found an effect, but was the effect really there? There may be problems with: - Reproducing results using the same data. - Replicating a result using new data. Because of errors or problems in analysis, results may not be reproducible even with the same data. For example, failure to account for cross-sectional correlation in financial return data can easily lead to standard-errors that are off by factor of 10. Because of what Andrew Gelman calls the garden of forking paths, all the choices a researcher must make can lead even a capable, honest researcher to find "statistically significant" results that aren't replicable in new data. In finance, it's not hard to find a trading strategy which through pure luck would have made obscene amounts of money in the past (and there are many people looking)! If I started a VC fund today that will only invest in startups where the co-founders are named Sergei and Larry, you might rightly question how well a back test is estimating expected returns. The numerous degrees of freedom one has when building trading strategies creates what Gelman would call a garden of forking paths. These are especially prominent problems in psychology but also pervasive throughout science. #### 2. The effect shrinks over time Another issue especially pertinent to trading (and much of social science) is that if an effect is based upon people making errors, there's the potential for people to wisen up once the effect becomes widely enough understood. If small stocks declined in December and advanced in January due to tax-loss selling and subsequent repurchasing, will those effects persist once more traders and investors are aware? In the finance context, distinguishing between (1) and (2) may be difficult (or practically, may not even matter). To some extent, everyone in macro-finance uses the same dataset: worldwide financial data. To obtain new data, we must all wait for the passage of time. #### References Gelman, Andrew and Eric Loken, "The garden of forking paths" Mclean, David R. and Jeffrey Pontiff, 2016, "Does Academic Research Destroy Stock Return Predictability?" Journal of Finance Peng, Roger D., 2009, "Reproducible research and Biostatistics," Biostatistics
Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.