Reducing Order Book Snapshot Storage with Periodic Checkpoints
Summary
The document addresses how to store order book states reconstructed by replaying add, cancel, and trade messages. Saving a full serialized state after every message can consume substantial disk space, especially when many consecutive states are similar. The stated research need is to retrieve the book near a chosen message quickly and generate later snapshots without replaying the full history each time.
The proposed compromise is periodic checkpointing: save a complete order book every fixed number of messages, then reconstruct an intermediate state by applying the messages since the nearest checkpoint. The checkpoint interval trades storage use against recovery time and should be chosen so that replaying the intervening messages remains fast enough. If the message stream can also be reversed, replay can start from either side of the target, reducing expected replay distance. The response gives a practical storage and access strategy, but no benchmark or universal interval; the right setting depends on the dataset and acceptable recovery latency.
Key ideas
- Saving a full state after every message can be wasteful when consecutive order books are redundant.
- Periodic complete snapshots can serve as checkpoints for reconstructing intermediate states.
- Choose the checkpoint interval to balance disk use against the time needed to replay messages.
- Reverse replay can reduce the expected number of messages needed to reach a target state.
Tags
Full text
# What is an efficient data structure to save order book states? # What is an efficient data structure to save order book states? First of all, I am aware of the highly related question What is an efficient data structure to model order book?, but my question is a bit different here. I want to save the order book states after the arrival of each message, which includes add, cancel and trades. For example, assume we're using a B-Tree with double-linked lists, after each message I'll serialize it to disk. These operations are done offline by replaying historical data, so I don't care too much about performance and I don't really need to implement the matching myself. My main purpose is to be able to fast recover what the order book looks like given some specific message and then generate some snapshot in future research, so that I do not have to replay all the data every time. Therefore, I'm mostly interested in how to effectively save these order book states to achieve low disk usage and fast loading. For some liquid instruments there could be millions of messages and it seems unrealistic to save all the snapshots, especially when they are highly redundant and serially correlated. I'm thinking about only saving the changes of order books as sequences as somthing like "price 100 + vol 1000, price 101 - vol 100", but then it would be path dependent. Any suggestions? ## Answer by Bob Jansen (score 3) https://quant.stackexchange.com/a/73325 One approach would be to make a snapshot every $n$ messages with $n$ chosen such that applying $n$ messages to a given orderbook is fast enough for you. This gives a reduction in storage requirements by a factor approximately $n$ and it should be quick to apply a limited number of messages. If you can play the messages backwards as well, the expected time to replay to a certain message halves.
Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.