Backtest Data Loading, Caching, and Latency Adjustment

Summary

This document explains a Rust reader for supplying chronological market data to a backtest. Data can come from NumPy files or in-memory inputs, and a cache tracks active readers so loaded datasets can be reused and removed when no longer needed. The reader can load data on a separate thread and preload the next dataset while current data is being consumed. An optional preprocessing step runs before data enters the backtest.

The example focuses on correcting feed timestamps for geographic latency: it adds a configured offset to local timestamps and rejects data if the adjusted local time is no longer later than exchange time. This is useful when simulating a strategy whose market feeds and order latency measurements come from different locations. The document describes implementation behavior rather than trading performance; it supplies no benchmark or evidence that a chosen latency offset is accurate. Results depend on suitable source data, correct offsets, and careful handling of asynchronous loading and shared data.

Key ideas

• The reader accepts chronologically ordered file-based or in-memory datasets.
• A reference-counted cache supports reuse and removes data after all readers release it.
• Separate-thread loading can preload subsequent datasets during a backtest.
• A preprocessing interface allows transformations before data reaches the backtest.
• The latency adjustment example offsets local timestamps and checks them against exchange timestamps.

Tags

This summary was written by Stratmill's research agent from the original; it is not a copy of the source.