A calendar year of 1-minute bars should be 525,600 rows. Our 2025 pull of BTCUSDT perps came back with 525,557 — 43 minutes short, a completeness rate of 99.992%. A mid-cap alt perp on the same venue, same pull, same code path, came back 1,247 short: 99.76%. And a US equity minute series for the same year had roughly 427,000 fewer rows than the crypto one, which is not a gap at all, just a market that closes.
Three of those numbers are boring. The one worth a thousand words is the 43.
Four different things that all look like a hole in your dataframe
Before the 43, the taxonomy, because most gap-handling code is wrong at this layer already. When a row is absent from your local parquet, you have no idea which of these produced it, and the correct response differs for each.
| Kind | What actually happened | How it arrives | Right response |
|---|---|---|---|
| No trades | Market was open; nobody crossed the spread that minute | Zero-volume bar (OHLC all equal, trades=0) on some endpoints, absent row on others | Keep it. It's real information: nobody wanted to trade. |
| Venue down | Matching engine offline, scheduled or not | Absent row, sometimes a burst of them | Mark stale. Do not trade across it. |
| Instrument halt | LULD band breach, news pending, delisting notice | Absent row, then a reopening auction print | Mark stale, and treat the reopen as a discontinuity. |
| Your fault | Rate-limited pagination, a retry that dropped a page, a timezone-boundary bug in the loop | Absent row, indistinguishable from the above | Detect and refetch. This is the one you can fix. |
Venues disagree on the first row of that table, which is the part that bites. Coinbase's candles endpoint skips empty buckets entirely, so an illiquid pair's history is full of literal holes. Binance's klines will generally hand you a synthetic bar with volume 0 and open=high=low=close pinned to the previous trade. Same underlying fact, two different shapes, and a loader that reindexes to a full minute grid turns one into the other without telling you. Check what your venue does before you write the gap-filler, not after.
The 1,247 missing minutes on the alt were almost entirely category one: a thin book at 04:00 UTC on a Sunday, nobody trading. Annoying, easy to reason about, and largely harmless because the strategy wouldn't have traded then anyway. Which is exactly why I stopped looking at it and went back to the 43.
The 43 minutes were not scattered
If missingness were uniform, 43 minutes across a year would land as 43 lonely singletons, one every eight and a half days, each one a rounding error. That's not what we got. They came in six runs: one of 19 consecutive minutes, one of 11, two of 4, and a couple of pairs. Six events, not 43 accidents.
And the events are conditional on the thing you care about. Venues go down when they're under load, and they're under load when price is moving. I bucketed every hour of the year by realized volatility and asked where the missing minutes lived: 61% of them sat in the top decile. The unconditional probability that a given minute is missing is 0.008%. Conditional on being inside a top-decile volatility hour, it's roughly 0.05% — six times higher, and clustered on top of that.
So the completeness metric on your data-quality dashboard is measuring the wrong thing. 99.992% sounds like a dataset you can stop thinking about. What it actually describes is a series that is complete during the hours your strategy does nothing and holed during the hours it does everything. A momentum system that fires on volatility expansion has a materially higher chance of hitting a gap than the headline number implies, and it hits the gap mid-trade.
What forward-fill does three lines later
Here's the failure that made me write this. Take the 19-minute run. Standard hygiene: reindex to the full minute grid, forward-fill OHLC from the last close, set volume to zero. The series is now continuous and your indicators run without a NaN in sight.
Those 19 bars have high == low == close. True range is zero for every one of them. An ATR(14) computed on that window, coming off a pre-outage reading of about 240 USDT, decays toward roughly 34 by the time the venue comes back — the five surviving real bars in the window carry the whole average. Now feed that into a volatility-scaled position sizer, the ordinary size = risk_budget / ATR kind. Size goes up sevenfold.
The next real bar is the reopening print, and it is not a quiet bar. In our case it opened 1.8% away from the last pre-outage close. The backtest happily took a 7x position into a 1.8% gap, on a fill that could not have existed, at a price nobody was quoting. That single synthetic trade was worth more than a month of legitimate P&L in the equity curve, in the wrong direction — and it was born entirely of a data-cleaning line written to make the dataframe tidy.
Dropping the rows instead of filling them isn't the fix either, it's the same bug wearing a different hat. Drop them and your integer-indexed lookbacks lie: a "20-bar EMA" now spans 39 wall-clock minutes across the outage, the bar-to-bar return over the boundary is the whole 1.8% jump treated as one minute of movement, and any per-bar volatility estimate reads it as a 60-sigma event. Nothing warns you. The index is still monotonic.
Resampling is where it goes invisible
Most research doesn't run on 1-minute bars, it runs on something aggregated, and aggregation launders the problem. Resample to 5-minute and a 19-minute hole becomes four bars, of which the first and last are partial. Pandas will compute a perfectly reasonable-looking OHLC from two surviving minutes and label it identically to a bar built from five. Nothing in the output distinguishes them.
The cheapest fix I know: carry a bars_in_window column through every resample and never throw it away. One integer per row, and every downstream question about whether a bar is trustworthy becomes answerable. We also carry seconds_since_last_real_print, which is the same information in a form the execution layer can act on.
The policy, such as it is
What our agents apply now, in order:
- Never reindex silently. The loader emits a gap manifest — start, end, length, and which of the four categories it believes it is. If a run of missing minutes is under 3 bars and volume in the surrounding bars is thin, it's a no-trade minute. Anything longer during active hours is treated as an outage until proven otherwise.
- Refetch before you interpret. Half our early gaps were pagination bugs. A second pull from a different endpoint or a different vendor resolves category four and shrinks the problem before any judgement is needed.
- Staleness gate, not gap-filling. The strategy gets a
data_ageinput and a hard rule: no new entries when the last real print is older than N bars, and open positions flatten at the reopen only via a market order priced with an explicit gap-risk haircut. Fills that couldn't have happened are worse than trades that didn't. - Indicators see NaN, not fiction. Forward-filled prices never reach the feature layer. If ATR can't be computed, it's undefined, and undefined means flat. Loud failure beats a quiet 7x.
- Report gap-conditional performance. Every backtest we ship shows P&L with outage-adjacent trades excluded alongside the headline. If those trades are carrying the result, the result is a data artifact.
A quick audit you can run today: group your missing minutes into consecutive runs, then check what fraction of your backtest's trades open or close within 30 minutes of a run boundary. Under 1% and the gaps probably aren't driving anything. If it's 5% or more, the equity curve is partly a story about exchange downtime.
The tell I've come to trust is the shape of the missingness rather than its size. A dataset with thousands of scattered holes in dead hours is usually fine. A dataset with a handful of tight clusters is telling you that something breaks under load, and the something that breaks under load is where your strategy lives.
← All posts
