Six weeks after we ran a backtest, we ran it again with the same code, configuration, and date range. The ending equity was different. Not wildly different, but enough to move the Sharpe ratio from 1.31 to 1.08 and change which parameter set looked best.
The strategy had not changed. The historical data had. A vendor had revised several days of trades and bars, and our first run had not preserved the exact input snapshot. We spent the next few days figuring out whether the new result was a data correction, a pipeline bug, or a research process we could not reproduce.
Monday: the rerun disagreed
The first clue was a small gap in a liquid spot pair. Our original bar showed a high of 62,418. The rerun showed 62,431. That difference seemed harmless until we traced the strategy's entry: its threshold sat between those prices. A few changed bars meant different signals, which meant different positions several days later.
We compared daily trade counts and found the mismatch started on a single date. Then we compared the raw rows feeding that date. The vendor's current response had a handful of corrected trades and a revised aggregate bar. There was no change log attached to the response we had used.
This is ordinary data behavior. Exchanges repair records; vendors backfill gaps, correct bad prints, and adjust aggregation logic. A query for “January data” can return a different answer in March.
Tuesday: we found the missing receipt
We had stored the vendor name, symbol, and date range in the run record. We had not stored the response itself or a content hash for each input partition. That meant we could prove what we asked for, but not what bytes the vendor returned.
We pulled the oldest cached files we could find and compared them with the current response. The difference was not a split adjustment or a timezone conversion. It was a small set of corrected trade records flowing through our bar builder. That ruled out one theory, but the cache was incomplete, so we still could not reconstruct every affected bar exactly.
| What we recorded | What it let us answer | What was missing |
|---|---|---|
| Vendor, symbol, date range | Which request the run made | The response content |
| Pipeline version | Which code built the bars | The precise input rows |
| Run timestamp | When we ran the job | Which vendor revision was available then |
We had treated “same query” as “same data.” That assumption had made the experiment look more reproducible than it was.
Wednesday: the correction changed the story
We replayed the affected dates using both versions of the available data. The revised bars removed two entries and moved one exit by a bar. Those three trades accounted for most of the Sharpe change. The strategy was sensitive to a few observations near its trigger, even though the overall price series looked almost identical on a chart.
That did not mean the original result was fraudulent or the revised result was automatically better. It meant the claim needed a scope: the old run described the data snapshot we had at the time, while the rerun described the vendor's current history. We could not honestly call one a reproduction of the other.
Thursday: we changed the research record
We added immutable input snapshots to new research runs. Each partition gets a content hash, and the run manifest records source, retrieval time, schema version, adjustment policy, and transformation code version. For large datasets, we keep the actual files in content-addressed storage, so identical partitions can be shared without overwriting prior versions.
When storage or licensing rules prevent us from retaining raw data, we preserve the strongest receipt the terms allow: hashes, request parameters, source revision identifiers, and a small sample of parsed rows for diagnosis. A hash cannot recover the data, but it can tell us when two runs did not consume identical bytes.
A reproducible backtest needs an input identity, not just a data URL. Record what arrived, when it arrived, and how it became the rows your strategy consumed.
Friday: what we would skip next time
We would skip comparing only headline metrics. A changed Sharpe tells you that something moved; it does not locate the cause. Start with row counts and partition hashes, then find the first differing input date. From there, compare transformed rows, signals, orders, and fills in that order.
We would also skip silently replacing old data with corrected history. Corrections are useful, but old runs should remain tied to their original inputs. A new dataset should create a new run, with a visible relationship to the older one.
Our practical rule now is simple: if a rerun is meant to reproduce an experiment, pin the data snapshot. If it is meant to benefit from corrected history, label it as a new experiment. The distinction took us a few days to learn, and it is one line in the run manifest.
← All posts
