Testing Strategy Robustness with Permutation, Bootstrap, and Monte Carlo Methods
Summary
The article presents a three-part statistical validation workflow for trade-level backtest results exported from MetaTrader. A sign-randomization permutation test asks whether a risk-adjusted statistic such as the Sortino ratio could arise without directional edge; independently flipping trade-return signs preserves their magnitudes while disrupting the assumed edge. Bootstrap resampling draws trades with replacement to estimate uncertainty intervals for performance metrics. Monte Carlo sequence shuffling preserves each trade's profit or loss but changes the order, producing alternative equity paths and drawdown estimates.
The methods answer different questions: permutation testing examines evidence against a no-edge null, bootstrap intervals assess estimate stability, and sequence shuffling tests sensitivity to trade order. The article also describes exporting one row per completed trade, grouping results by indicator variant, and combining results in a report. These checks characterize historical data under specific assumptions; they cannot establish future profitability, anticipate regime changes, or account for execution costs and overfitting. The article positions passing the battery as a reason to continue with out-of-sample evaluation, not as a final deployment guarantee.
Key ideas
- Sign-randomization tests whether observed risk-adjusted performance exceeds a null model with no directional edge.
- Bootstrap resampling estimates the uncertainty of performance metrics across alternative samples of trades.
- Trade-sequence shuffling preserves trade outcomes while testing how their order affects equity curves and drawdowns.
- The three methods test distinct properties and should not be treated as interchangeable.
- Historical robustness results do not guarantee future profitability or account for all trading frictions.
Tags
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.