Backtesting System Design, Biases, and Realism Tradeoffs
Summary
This article explains why backtests are models of past trading rather than forecasts of live performance, and compares simple bar-by-bar loops with event-driven systems. A loop-based tester is quick and easy to build for initial strategy screening, while an event-driven design routes market, signal, order, and fill events through components closer to a live trading workflow. Greater realism generally comes with more implementation complexity.
It catalogs sources of misleading results, including in-sample overfitting, survivorship and look-ahead bias, changing market regimes, omitted transaction costs, imperfect OHLC execution assumptions, capacity limits, benchmark choice, sensitivity to start dates, and drawdown tolerance. The article recommends realistic costs and data, robustness checks, and skepticism toward unusually strong results. It advocates building a backtester as a learning exercise, but the source excerpt is incomplete and does not provide a full implementation or empirical comparison of system types.
Key ideas
- A backtest filters strategy ideas by applying rules to historical prices, but it cannot reproduce live trading exactly.
- Loop-based systems are simple and fast, while event-driven systems more closely resemble live execution infrastructure.
- Look-ahead bias, survivorship bias, overfitting, and regime changes can distort simulated performance.
- Costs, market impact, capital limits, benchmark choice, start-date sensitivity, and tolerable drawdowns matter to realistic evaluation.
- Strong backtest results warrant further scrutiny before capital is committed.
Tags
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.