Why Market Understanding Matters More Than Unconstrained Data Mining

Summary

This essay argues that systematic traders should begin with market observation and an explanation of why a possible edge exists, rather than searching broadly across indicators and parameters for a profitable backtest. Repeated experimentation can produce apparently strong in-sample results and even pass out-of-sample or robustness checks when the number of attempted tests is large and undisclosed. Such results may fail in live trading, while leaving the trader unable to explain the source of performance or judge when it has stopped working.

The suggested process is to form a market-based hypothesis, investigate its structural or behavioral cause, and use simple analysis aimed at disproving it. Traders can then implement plausible edges, learn from live feedback, and refine their approach. Data mining can still help when prior market knowledge constrains the search space and results receive careful sanity checks. The article offers a practical research philosophy rather than empirical comparisons or a formal statistical procedure, and it does not specify controls that guarantee against data mining bias.

Key ideas

• Broad parameter searches can uncover backtests that look profitable by chance.
• Undisclosed multiple testing makes out-of-sample and robustness results harder to interpret.
• Start research by observing markets and proposing a structural reason for a possible edge.
• Use simple analysis to challenge a hypothesis instead of only seeking confirmation.
• Constrain data mining with market knowledge and careful sanity checks.

Tags

This summary was written by Stratmill's research agent from the original; it is not a copy of the source.