Why Short Samples Cannot Reliably Detect a Trading Strategy's Decline
Summary
The article examines why a few weeks of weak performance cannot establish that a strategy has lost its edge. In a simulation, a strategy with positive long-run drift shifts to zero drift for one month while volatility remains high. A comparison of the five-year and one-month samples fails to detect the change: the reported Welch test and Kolmogorov–Smirnov test are not significant. Simulations also show that a zero-drift month can post a strong return by chance, making performance misleading in either direction.
The discussion explains that small return signals are overwhelmed by daily noise, and that distribution tests may have even less power than mean tests when the change is a small shift in average return. It argues for treating confidence in an edge probabilistically and adjusting exposure as evidence changes, rather than relying on a binary significance test. The examples use a simplified return model and do not provide a universal decision rule; real markets may have changing volatility, tails, and dependencies.
Key ideas
- A short period of underperformance may be too noisy to distinguish from ordinary variation.
- A strategy with zero expected drift can still produce an unusually strong month by chance.
- A Kolmogorov–Smirnov test may have low power for detecting a small mean shift when variance is unchanged.
- Statistical tests based on short samples may lag behind changes in a strategy's edge.
- Bayesian belief updates and gradual exposure changes offer a probabilistic alternative to a binary dead-or-alive decision.
Tags
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.