A frozen market-data snapshot is not enough to make a backtest trustworthy. It can reproduce the same answer while hiding that the vendor changed the inputs that produced it.
Keep the snapshot. Then keep a change log that can tell you what moved, when it moved, and whether the strategy cared. This matters most when your research uses corrected trades, revised fundamentals, reconstructed funding, or a vendor's quietly refreshed history.
Why can a reproducible backtest still mislead you?
Suppose you download five years of hourly bars on Monday and run a strategy. On Friday, the vendor has repaired a bad trade, adjusted a corporate action, or filled a gap. Your saved file still produces Monday's result exactly. That is reproducible. But if you rerun from the vendor's current history, you may get a different result and have no record of why.
The problem isn't limited to dramatic corrections. A single altered high can trigger a stop; a revised close can move an indicator across its threshold. In a broad universe, one symbol's history may change its rank and push another asset out of the portfolio. The final Sharpe can drift even when the strategy code and configuration are identical.
A file hash tells you that two files differ. It doesn't tell you whether a split adjustment changed 40 years of prices, a vendor corrected three trades, or a parser turned an empty field into zero.
What belongs in a market-data change log?
Record changes at the smallest useful unit, with enough context to replay the affected research. For a bar feed, that usually means the instrument, interval, event time, prior and new values, source, and the time you observed the update. Preserve the original payload when the vendor supplies one.
| Record | Why keep it | Example |
|---|---|---|
| Source version or retrieval time | Identifies the vendor state used | Snapshot fetched 2026-11-02 08:15 UTC |
| Changed field and old/new values | Shows what actually moved | ETHUSDT 1h high: 3,412.10 → 3,418.60 |
| Change classification | Separates repairs from transformations | Late trade correction, split adjustment, backfill |
| Pipeline and schema version | Distinguishes source edits from your own processing | Normalizer v7, adjusted-bar schema v3 |
For large datasets, store immutable snapshots and a compact manifest of their checksums, source versions, and retrieval times. Add row-level diffs for changed partitions. That gives you a practical middle ground: full replay remains possible without writing every unchanged row to a second audit table.
How do I measure whether a correction matters?
Don't stop after counting changed rows. Re-run the affected strategy with the old and new data under the same code, parameters, fees, and execution assumptions. Compare the decisions and the path to the result.
- Identify affected instruments and event times from the diff.
- Find which features, signals, orders, or portfolio weights consume those records.
- Replay the smallest valid window, including any lookback and open position state.
- Compare orders, fills, exposure, turnover, drawdown, and final performance.
For example, a vendor correction that changes a 1-minute high by $0.02 may touch a million rows and have no effect on a daily momentum strategy. One corrected bar at a stop level can matter more. Measure the decision impact; a count of changed records is a poor proxy for it.
Keep two questions separate: “Can I reproduce the result from the data I saved?” and “Would this data version have been available to the strategy then?” A change log helps answer both, but the second also needs point-in-time availability records.
What does this cost, and when is it worth it?
Change tracking adds storage, ingestion work, and operational decisions. Vendors may not expose stable versions, and some historical feeds are periodically rebuilt without a clear correction notice. You can't log a change you never observe.
Start where revisions can alter research conclusions: bars used by stop logic, corporate-action-adjusted equities, point-in-time fundamentals, and funding or open-interest series that get reconstructed. A daily manifest plus partition checksums may be enough for broad, stable datasets. Add field-level diffs where updates are common or trade decisions are sensitive to individual records.
Critics are right that a complete, perpetual archive of every vendor response can become expensive and still fail to capture what the source knew at the time. The answer is to scope the audit to the data and decisions that matter, and document the gaps. A snapshot lets you rerun an experiment. A change log lets you explain why the experiment's answer moved.
← All posts
