Same commit, same parquet files, same machine, two runs an hour apart. Sharpe 1.34 and Sharpe 1.19. Total return 41.2% and 37.8%. Trade count 1,418 and 1,421. And the two equity curves agreed to every printed decimal for 11,205 consecutive bars before they split.
The first three numbers are annoying. The last one is the interesting one, because it says the problem isn't sloppy floating point smeared across the whole run. Something discrete happened at one specific bar, and the rest was compounding. This post is about finding that bar, and about what the size of that 0.15 does to every parameter sweep you've ever run.
The strategy: cross-sectional momentum on the 30 highest-volume USDⓈ-M perps, 4-hour rebalance, long the top five by 12-hour return, short the bottom five, 18 months of history, fees and funding charged per leg.
Finding the bar where the runs part ways
If you log per-bar equity at full precision, this takes about four minutes. Dump both runs to a CSV of (bar_index, equity, open_positions_hash), load both, and find the first index where they disagree. We'd already written that script for a different bug, which is the only reason I didn't spend a morning on it.
Bar 11,206, 2025-03-14T08:00 UTC. Equity identical at the previous bar to 13 significant figures. At 11,206 the position hashes differ: run A is long SOL, run B is long AVAX. Same side, same notional, different symbol. Everything after that is downstream.
So I printed the ranking inputs for that bar in both runs. They were identical. Byte-for-byte identical, same 30 symbols, same 30 scores. Two of the scores were 0.0. Exactly zero, both of them, because both names had printed a 4-hour bar with zero trades inside the lookback window, so close-over-close came out to one, and the log came out to zero. Not rounding-to-zero. Zero.
Two symbols tied at rank five. The selection took the top five. Which of the two got in depended on the order the sort left them in, and the sort wasn't doing what I assumed.
A tie, an unstable sort, and the thing I'd been ignoring for two years
The ranking ran through a grouped aggregation, and the frame feeding it came from a dict comprehension over a set of symbols that was rebuilt per bar from an async fetch. The set's iteration order shifts with the hash seed, and Python randomizes the string hash seed per process unless you pin PYTHONHASHSEED. So the pre-sort row order differed between runs, and with a non-stable sort on a tied key, the tie broke differently.
Ties like this aren't a freak event. They're structural. Anywhere a feature saturates or clips, you manufacture exact equality: zero-volume bars give exactly-zero returns, a clipped z-score pins at ±3.0, a rank-transform with few distinct values produces dozens of ties, a boolean filter scores everything that passes at 1.0. Over 18 months at 4-hour bars this run had 47 bars with a tie at the selection boundary. Three of them changed the selected basket. The rest tied between two names that were both already in or both already out.
For about a day I was convinced the data loader was nondeterministic, because that's the exciting answer. It wasn't. It never is. It's a set, a tie, and an assumption about sort stability nobody wrote down.
Why three trades are worth 0.15 Sharpe
This is the part people push back on, and the answer is that a backtest with equity-fraction sizing is a path-dependent system. Sizing at 8% of current equity per leg means a difference in equity at bar n is a difference in every notional from bar n forward.
The first divergence cost very little on its own. Run B's AVAX leg lost 2.1% over nine hours; run A's SOL leg gained 0.4%. Equity gap after that trade: 0.21%. Trivial. But from there the two runs are no longer the same strategy. They hold slightly different size, so they cross slightly different funding accruals, and two of the later boundary ties broke differently again because the scores feeding them now came from marginally different held positions. One of those landed on 2025-03-27, a day before a six-day trend that produced roughly a third of the run's total PnL. Run A was on for the whole move; run B entered one rebalance late.
Return gap: 3.4 points. The Sharpe gap is bigger than the return gap implies because run B's reordered trades overlapped a more volatile stretch, so the denominator rose while the numerator fell. Small cause, two amplifiers.
If your sizing is fixed-notional and your entries don't depend on current holdings, you're much more insulated. Most interesting strategies are neither.
The five places it actually gets in
| Source | Symptom | Fix |
|---|---|---|
Unpinned PYTHONHASHSEED with set/dict iteration order feeding a sort | Tie-breaks flip between runs; first divergence at a specific bar | Pin the seed; sort by an explicit secondary key (symbol) so ties are deterministic |
Non-stable sort on a tied key (quicksort default in NumPy/pandas) | Same as above, survives seed pinning | kind="stable", or make the key total |
| Unseeded RNG in bootstrap, train/test shuffles, or synthetic fill jitter | Whole-run drift, no clean divergence point | One explicit seed per component, logged in the run manifest |
| Parallel float reduction (thread-count-dependent summation order) | Differences in the last few bits, usually harmless until they cross a threshold comparison | Pin thread counts for research runs; never compare floats with == at a decision boundary |
| Unpinned library versions | Reproducible today, not in November | Lockfile hash in the manifest alongside the data snapshot hash |
The fourth row is the one that doesn't matter as often as people fear, and the first row is the one that bites constantly.
Bit-reproducible is a tool, not a virtue
You want determinism so that when you change one line, the diff in the equity curve is attributable to that line. That's the whole reason. Every agent run on Stratmill now writes a manifest with the data snapshot hash, the lockfile hash and every seed, and a rerun that doesn't reproduce the prior curve bit-for-bit is a failed build, not a curiosity.
But once you can reproduce, deliberately break it. Run the thing 64 times with 64 seeds and look at the spread:
The jitter band. Same strategy, same data, 64 seeded tie-break and fill-order permutations. Sharpe p5 1.12, median 1.27, p95 1.41. Band width 0.29.
Now go back to the parameter sweep. Best config scored 1.46. The config sitting 40th out of 96 scored 1.31. The gap between them is 0.15, which is half the band. The sweep didn't rank those two configs. It sampled one draw from each of their distributions and sorted the draws.
That reframing changed how we pick. A sweep result is only a ranking if the gaps between configs exceed the jitter of a single config, and on a path-dependent strategy with 1,400 trades the jitter is usually large enough to flatten the top third of the leaderboard into a tie. When that happens, pick on something the band can't hide: lower turnover, fewer parameters, a cost assumption you'd defend to a skeptic, better behaviour in the walk-forward fold you like least. Those are real tiebreakers. A 0.15 Sharpe edge is not.
One more thing worth doing before you trust any of it. Run your backtest twice right now, diff the per-bar equity, and find out whether you're in the bit-identical camp or the 0.15 camp. It's a fifteen-minute experiment and it tells you how much of your research history was measuring the strategy versus measuring a hash seed.
← All posts
