Testing Trading Strategy Returns for Statistical Significance

Summary

This article presents a toolkit for assessing whether observed trading returns are distinguishable from chance. It explains why net profit, profit factor, and win rate do not convey uncertainty, and describes one-sample and Welch t-tests, Mann–Whitney U tests, confidence intervals, and comparisons between high- and low-volatility regimes. The proposed implementation is in MQL5 and reports statistics and plain-language interpretations.

The toolkit accepts either log returns built from market prices or equity-normalized realized P&L from closed deals. These sources support different conclusions: price data can test market drift or regime differences, while deal data tests a strategy’s realized P&L stream. The article presents t-tests as useful with sufficiently large samples but notes their sensitivity to return distributions; Mann–Whitney provides a rank-based complement that is more robust to skew and outliers. Statistical significance does not establish that an edge will persist live, and failure to reject a zero-mean hypothesis on a short sample does not prove a strategy is worthless.

Key ideas

• Positive net profit, profit factor, and win rate do not measure the likelihood that results arose by chance.
• The toolkit pairs mean-based t-tests with the rank-based Mann–Whitney U test for sample comparisons.
• Price returns and equity-normalized deal P&L answer different statistical questions.
• Volatility regime splits can test whether performance differs across market conditions.
• Significance is evidence about a sample, not a guarantee of future live performance.

Tags

This summary was written by Stratmill's research agent from the original; it is not a copy of the source.