Backtesting Limits: Execution Bias, Overfitting, and Survivorship
Summary
The article explains backtesting as a way to replay trading rules on historical data, check whether signals and logic behave as intended, and expose strategy defects before deployment. It emphasizes that historical simulation can test market assumptions, but cannot by itself show that a strategy will earn money in live markets. Examples distinguish unstable signals on unfinished bars from future-data leakage, and describe “stealing price” cases where a simulator grants fills at prices unavailable after a trigger.
Other traps include ignoring exchange price limits, queue priority, partial fills, one-sided execution in arbitrage, extreme liquidity gaps, and market impact or slippage. The discussion also warns that strategies with many tuned parameters may fit limited historical samples, while selecting only surviving funds or the best result from many trials exaggerates apparent skill. It recommends realistic slippage assumptions and paper trading to compare simulated behavior with live conditions. The examples are conceptual; the article supplies no systematic test results or universal method for validating a strategy.
Key ideas
- Backtesting checks historical signal behavior and strategy logic, but cannot establish future profitability.
- Unfinished-bar signals and future-dependent indicators can create signals that disappear in real time.
- Simulated fills can be unrealistic when they ignore gaps, price limits, queue priority, partial execution, or slippage.
- Repeated tuning on limited data and selection of surviving or best-performing strategies can exaggerate results.
- Paper trading and more realistic execution assumptions can help assess whether simulated behavior carries over.
Tags
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.