Why AI Backtests Need Economic Theory and Statistical Discipline
Summary
The article argues that AI makes it easy to generate and test trading rules, but that speed also encourages data mining. Repeatedly changing parameters, filters, timeframes, or asset universes amounts to many hypothesis tests; a strong historical result can emerge by chance. The document uses the multiple-testing problem to explain why a polished backtest is weak evidence when many experiments have been tried.
It recommends starting with an explanation for why a trade should earn a return, such as a risk premium, liquidity provision, behavioral bias, or institutional constraint. Momentum is offered as an example of a pattern that may have structural causes, while parameter optimization without such a theory risks fitting noise. The author sees AI as useful for technical work but insufficient for judging whether an apparent edge is economically plausible. No strategy performance study is presented, so the piece is guidance on research process rather than evidence that any particular edge works; sound theory and disciplined validation still cannot guarantee profits.
Key ideas
- AI lowers the cost of strategy experiments and therefore increases the chance of false discoveries.
- Repeated optimization and rule changes create implicit hypothesis tests that can fit noise.
- A backtest is more informative when a plausible market mechanism explains why the strategy may earn returns.
- Potential sources of edge include risk premia, liquidity provision, behavioral biases, and institutional constraints.
- AI can speed up implementation, but researchers must supply skepticism and statistical judgment.
Tags
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.