Order Book Recovery with Asynchronous Event Persistence
Summary
The document considers how to recover a limit order book after a process or machine failure. The proposed approach in the question is to save the full book state periodically, while the reply focuses on persisting the event stream so the system can rebuild its state. It recommends placing writes off the matching engine’s critical path, using a queue and an asynchronous persistence worker.
The reply says modern database systems can handle high insert rates, but committing every event individually is generally inefficient; buffering and batch commits are suggested. Other options include redundant hosts, flat files, high-throughput databases, or persistent messaging systems. For stronger capture guarantees, it recommends recording incoming order messages at the network edge, before updating the book, with examples of capture hardware and commercial messaging platforms. The appropriate design depends on operational requirements. The document does not provide benchmarks, recovery-time targets, or a detailed comparison of durability and failure modes.
Key ideas
- An asynchronous writer can persist queued events outside the order matching engine’s critical path.
- Batching events can make database persistence more efficient than committing each event separately.
- Recovery can use databases, files, redundant hosts, or persistent messaging systems.
- Capturing incoming messages at the network edge can preserve them before book updates occur.
- The suitable persistence design depends on system and business requirements.
Tags
Full text
# Limit Order Book disaster recovery technique # Limit Order Book disaster recovery technique I have designed a fast limit order book, similar to the one described here. It is working and passing the unit tests. Now, I am thinking of its reliability. i.e in case of failure (e.g unhandled exception or machine restart etc) I have sketched a design, that after every hour, we store the whole state of the orderbook in a Database. Then in case of failure, we can recover the last hour's state and continue it. In a order matching engine with >>1000 Transactions/Second (TPS) , is this even feasible to use a Database for disaster recovery of an orderbook? ## Answer by databento (score 6) https://quant.stackexchange.com/a/68688 For data on a given single host, a common architecture is to write the data onto a queue (could be multi-consumer or single consumer), and have an asynchronous writer that reads it off and persists it in parallel to the critical path operation. Beyond that, it doesn't quite matter how you persist your order book for recovery. There's many "OK" ways to do it and the optimal design will depend a lot on other business requirements. It is possible to accommodate 1000 transactions per second and more with today's hardware on a typical DBMS. But that said, committing event-by-event to the database wouldn't be a very conventional or efficient pattern. If you intend to store them in a database ultimately, you will likely want to buffer up the events somehow and commit them in batch. You could do that in-memory with redundant hosts; with a flat file on host; with a more modern, likely distributed, database management system designed for high rate of inserts per second; or with a messaging framework that has persistence like Kafka - really anything. If you really want to do it correctly though, I would recommend looking into hardware-driven solutions such as an optical tap; port mirroring; a layer 1 switch; a dedicated capture card, or commercial messaging solutions such as Solace, Informatica or Tibco. Persist the incoming order messages as soon as they arrive on the edge of your network, even before you update the order book.
Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.