We restarted a paper strategy halfway through a trade and got a different order sequence from the one it had produced before the restart. Same market data, same strategy version, same account balance. The signal calculation matched. The strategy's memory of its position and pending orders did not.
We traced the mismatch over a day of replay work. The useful lesson was simple: a restart is a market event for your software. If you don't test how the strategy rebuilds its state, a clean backtest can conceal a paper system that forgets what it owns.
09:10 — We picked a boring position
The test strategy traded a liquid perpetual: enter long when a short moving average crossed above a longer one, then exit on the reverse cross. We started the replay with a small position already open and a reduce-only limit order resting to trim it. That gave recovery two facts to reconstruct: what we owned and what we had already asked the venue to do.
At the checkpoint, the account held 0.04 contracts. The order was open for 0.01. The strategy process had cached both values in memory, but its startup path only fetched the position. It treated the cached order as gone.
09:25 — The first duplicate order appeared
On restart, the strategy saw the existing long, ran its signal logic and submitted another 0.01 trim order. The paper venue now had two live orders. Neither was a bad order by itself. Together, they could sell twice the intended amount if both filled.
We initially blamed the signal loop. It wasn't the loop; it was the incomplete snapshot. The strategy asked, “What position do I have?” but never asked, “Which orders are still working?”
| State after restart | What the process believed | What the account held |
|---|---|---|
| Position | Long 0.04 | Long 0.04 |
| Open trim orders | None | Two for 0.01 each |
| Intended exposure after one fill | Long 0.03 | Could become long 0.02 |
10:00 — We fixed recovery, then found a timing edge
We changed startup to rebuild state from the account's positions and open orders before enabling new decisions. That removed the duplicate. Then we made the disconnect less tidy: one order filled while the strategy was offline, and the fill notification arrived after reconnect.
The account snapshot already reflected the fill. The delayed notification then reduced the local position a second time. For a few seconds, the strategy believed it held 0.02 contracts when the account held 0.03. Its next rebalance was based on a phantom shortfall.
We added reconciliation rules: treat the account snapshot as the starting point, use event identifiers to ignore fills already reflected there, and don't submit orders until the initial sync finishes. A notification can arrive late or twice. Recovery has to tolerate both.
13:40 — The replay caught a quiet mismatch
We replayed the same price path across the original run and the restarted run. Comparing final P&L would have missed the problem: the two versions ended with the same position after the market reversed. Comparing order events exposed it.
We logged each decision with the state it read: position, open orders, last processed fill identifier, signal value and strategy version. The first divergent event then had an explanation. One run saw a working order; the other saw an empty list. Later, one applied a fill twice.
A matching ending balance doesn't prove matching behavior. Compare the sequence of decisions and orders, especially around recovery boundaries.
16:20 — What we'd do differently next time
We had spent too long replaying price data before checking the account state transitions. Next time, we'd inject the failure cases first and keep the market path almost flat. That makes the software error easy to see, without a volatile move muddying the diagnosis.
- Restart with a position and a partially filled order.
- Disconnect after submission, then reconnect before the fill notification arrives.
- Deliver the same fill event twice and verify it changes state only once.
- Block new orders until positions and open orders have both reconciled.
- Compare decision and order logs across uninterrupted and restarted runs.
We also learned to save a recovery snapshot alongside the strategy version and event log. It made the failure reproducible in minutes instead of relying on someone to remember the exact reconnect sequence.
A paper strategy that behaves correctly only while its process stays alive hasn't passed a full rehearsal. Restart it mid-position, make fills arrive late, and inspect every order it sends afterward. The point is not to prove it never fails. It's to make its recovery visible before the paper account has to teach you the same lesson by surprise.
← All posts
