There's a question every backtest dodges: would the same code, fed the same market live, have done the same thing? You can't answer it with more backtesting. You can only answer it by running the strategy against live data with real execution semantics, then checking — trade by trade — that the two worlds agree. We do that check every day, automatically, and it has been the single most productive bug-finder in the platform.
Day one: build the mirror
Each promoted strategy runs in its own isolated live node, consuming real exchange streams: live quotes for fills, real spreads, real session hours. No shared state between strategies (we learned that one the hard way — a shared node let one strategy's position queries bleed into another's). Every fill is recorded with the signal value that produced it.
Every morning: replay and diff
A daily job takes each strategy's paper window, runs the identical code through the backtest engine over the same period on catalog data, and matches fills one-to-one: same side, same timestamp tolerance, same size. The output isn't a correlation or a vibe. It's a table: matched, paper-only, replay-only. A healthy strategy sits at 54 of 55 matched. An unhealthy one tells you exactly which trade diverged and when.
What broke, in the order it broke
- Warmup fills. The live node fed history to warm up indicators — and strategies with custom bar handlers happily traded on it. Backtests never saw those phantom entries. The fix was a submission-layer guard, not a strategy fix, because every strategy inherits the bug otherwise.
- Sequential multi-leg warmup. A cross-sectional strategy warmed up one instrument at a time; every leg saw empty siblings and the anchor never initialized. Live, it traded nothing; replayed, it traded fine. Zero matched trades — but only the diff said so out loud.
- Tick-precision crash loops. Live aggTrade ticks arrive at a different precision than the sandbox instrument expected. The node crash-looped on an assertion the backtest could never trigger, because catalog bars are already normalized.
- Restart amnesia. Process restarts re-fired entry logic on positions that already existed — for a rebalancing strategy, a restart quietly became a rebalance. In live trading that's not a parity footnote; that's a red flag with money attached.
Notice the pattern: not one of these is a signal bug. They're all execution-boundary bugs — warmup, state, precision, restarts. That's exactly the class of failure backtests are structurally blind to, because the backtest engine is the execution boundary in a backtest.
The asymmetries we keep on purpose
The two worlds are deliberately not identical. The backtest charges a modeled half-spread plus square-root impact inside PnL; paper pays the real spread in the fill price and no impact model. Adding the modeled spread to paper (or the real one to backtests) would double-count. Parity means same decisions, same fills within tolerance — not bit-identical PnL. Knowing which differences are principled and which are bugs is most of the discipline.
A backtest is a hypothesis. Paper trading is the experiment. Replay parity is the lab notebook that catches you fooling yourself.
If you run strategies and you're not diffing live behavior against replayed behavior on a schedule, you have a category of bug you've never seen. We find one roughly every couple of weeks, and each one was invisible from inside the backtest.
← All posts
