A perfectly reproducible backtest can be less trustworthy than one that moves a little. Repeatability proves that you can get the same answer twice; it says nothing about whether the answer depends on one arbitrary data snapshot, a library quirk, or a hidden default.
That sounds like an argument against reproducibility, and it isn't. Freeze the run so you can explain it. Then deliberately vary the assumptions that ought not to decide the result. The first step makes a result auditable; the second tells you whether it means anything.
What does a reproducible backtest actually establish?
If the same code, data, configuration, and random seed produce the same trades and metrics, you have controlled an important source of confusion. An agent's “improvement” can be compared with its parent run. A bug fix can be separated from a changed fee schedule. A colleague can inspect the same ledger and find the same fills.
But the result is conditional on everything you froze. Consider a crypto strategy whose historical file contains one-minute bars. You save the file hash, pin the code, and get a Sharpe of 1.4 on every rerun. If the vendor later corrects several candles, that clean reproducibility still describes the old file. It does not tell you whether the signal survives the corrected history, or whether those candles were available in that form at the time.
The hash is useful: it makes the dependency visible. It isn't evidence that the dependency is harmless.
Keep the run fixed; challenge the finding
I like to separate two questions. Can we reconstruct this exact run? And does the research conclusion survive reasonable changes? The first calls for an immutable record; the second calls for controlled perturbations, one at a time.
| What to record | What to vary afterward | What it can reveal |
|---|---|---|
| Raw-data version and checksums | Vendor revision or a second source | Dependence on corrected or vendor-specific values |
| Code commit and package versions | Numerically equivalent implementation | Order, rounding, or library sensitivity |
| Fees, funding, and fill assumptions | A defensible cost range | Whether a thin edge needs unusually kind execution |
| Universe and point-in-time membership | Alternate eligible-market rules | Dependence on inclusion or exclusion decisions |
For example, suppose a futures strategy shows 8% annualized excess return in a baseline replay. Re-run it with taker costs 25% higher, funding shifted to the next observed settlement, and the two largest contributors removed in turn. Those are not three ways to improve the model. They are probes: each asks whether a plausible detail is carrying the headline.
Keep each probe distinct. If you change the data, fee schedule, and execution timing together, a worse result gives you little diagnosis. And if a probe is implausible, label it as a stress case instead of presenting it as the likely outcome.
Don’t mistake byte-for-byte sameness for numerical truth
Some computations are sensitive to order. Floating-point addition is not associative, so summing many small P&L values in a different order can change the last few digits. Parallel reductions, hardware, and library versions can all affect exact output. Usually that is harmless; sometimes a tiny numerical change flips a threshold, changes a position, and sends the rest of the simulation down a different path.
That is why I’d preserve both the exact artifacts and tolerances for comparisons. Record the trade ledger, not just the final Sharpe. Compare decisions and cash flows first, then metrics within a stated tolerance. If a one-cent difference changes whether an order is submitted, inspect that boundary explicitly.
A useful run record includes the data and code identity, configuration, random seed, execution environment, and enough intermediate output to trace a metric back to fills and cash flows. A final score alone is not a reproducible research artifact.
When variation is noise, and when it is a finding
Not every changed number is a crisis. A tiny metric drift from a different reduction order may leave the trades unchanged. A changed trade near a threshold may be worth reviewing. A strategy that loses most of its result when one vendor revises a small patch of history has learned something about its own fragility, even if the original run remains perfectly reproducible.
Critics of robustness checks are right about one thing: it is easy to keep changing assumptions until a preferred strategy looks resilient, or to reject a sound result because some deliberately extreme scenario breaks it. Predefine the plausible variations, report the whole set, and distinguish ordinary alternatives from stress cases. Then preserve the baseline so another researcher can reproduce exactly what you saw.
Reproducibility gives you a dependable starting point. The useful question comes next: which parts of the answer stay put when the world around that run changes in ways you could reasonably expect?
← All posts
