Compressing Tick Data with Price and Time Differences
Summary
The document describes a tick-data compressor that stores changes in bid, ask, and time rather than repeating full tick records. Small price and time changes can fit into a compact representation, while larger differences use additional bytes. It also offers a three-byte mode, optional ZIP compression, and formats that retain volume, last price, and flags. The notes explain block-based compression and decompression, including the memory and speed tradeoffs of processing one large block versus smaller blocks.
Reported tests give byte-per-tick and throughput comparisons for BTCUSDT and EURUSD. The best representation varies by instrument: the share of ticks fitting the smallest encoding differs, and ZIP changes the resulting sizes. These are implementation benchmarks rather than trading-performance evidence. Results depend on instrument tick statistics and settings; the document recommends measuring the data before choosing a compact encoding. It also notes that restoring flags adds generation time, and that very large blocks can slow decompression and require more memory.
Key ideas
- The compressor encodes price and timestamp differences from the previous tick to reduce storage.
- Small changes can use compact encodings, while larger changes expand to multiple bytes.
- A three-byte mode may be preferable depending on an instrument’s tick distribution.
- ZIP compression can further reduce storage, with results varying across instruments.
- Block size affects memory use and decompression speed.
Tags
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.