Random Seeds and Reproducibility in Machine Learning Backtests

Summary

The document describes a backtesting question in which repeated runs of the same classification model and data produce sharply different returns. It reports that Random Forest and other scikit-learn models produced a gain in one run and a loss in another, raising concerns about result variability. The accepted response recommends setting the model’s random state so the algorithm’s random choices are repeatable across runs.

This is a basic reproducibility measure, not a method for making a strategy profitable or reducing genuine uncertainty in its performance. Fixing a seed helps isolate stochastic variation during experimentation, but the document does not investigate other sources of variation, assess out-of-sample performance, or compare models and backtests. Its advice is brief and should not be read as evidence that a more stable simulated result will generalize to live trading.

Key ideas

• Randomized classification algorithms can yield different backtest results across runs.
• Setting a model’s random state makes its random choices repeatable.
• Repeatability does not establish that a strategy is profitable or robust.
• The document does not analyze other sources of backtest variability.

Tags

# ML classification algorithms give random profit


# ML classification algorithms give random profit












I use `backtrader` `python` framework to backtest `ML` classification algorithms to make decision to buy or to sell.

When I use `RandomForest` or other algorithms in `scikit-learn` packages it gives up to 55% of profit:

The next run of absolutely the same code and data (just next run) gives 22% of loss:

Why is that? And what are the methods to avoid such a big range of results? Less, but more stable profit is better :)

## Answer by 2math (score 3, accepted)

https://quant.stackexchange.com/a/32424

Set the random_state = 0 as a parameter in the model and retry this.

Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)

This summary was written by Stratmill's research agent from the original; it is not a copy of the source.