Why Historical Tick Data Quality Varies Across Vendors

# Does the Integrity of Tick Data Vary by Vendor?


# Does the Integrity of Tick Data Vary by Vendor?












My firm is currently looking at various vendors to purchase historical tick data from.

The purpose of this question is not to ask for opinions on the best vendor, but purely on whether there is a substantive difference in the integrity of the tick data by vendor.

So everyone's on the same page, let's define integrity as being a measure of what proportion of ticks are "sane" i.e. would need to be cleaned or thrown out in actual use.

Assuming that you were paying for a vendor's complete tick database, is all the data essentially the same in the end regardless of vendor (as presumably they're all aggregating it from the same sources), or do different aggregation methods result in (statistically significant) different levels of accuracy?

Would this accuracy difference get washed out if you aggregated to e.g. the nearest microsecond?

You can provide specific examples if you want, though the question is intended more to answer whether someone looking at a tick data vendor should be evaluating the underlying quality of the data in addition to e.g. price and API capabilities.

## Answer by databento (score 11, accepted)

https://quant.stackexchange.com/a/64272

> whether someone looking at a tick data vendor should be evaluating the underlying quality of the data in addition to e.g. price and API capabilities.

Absolutely, you should evaluate the quality of the data. Even as of today (2021), there's plenty of data integrity issues that get passed on from a vendor to you, and make material differences to your use case.

You can generalize them into 4 types of issues:

- Packet loss and session restart issues

- Parser errors

- Lossy normalization

- Timestamping issues

## Packet loss and session restart issues

> [...] is all the data essentially the same in the end regardless of vendor (as presumably they're all aggregating it from the same sources)

They're not the same despite sourcing from the same raw feeds.

Many modern venues implement their raw feeds over UDP transport, which is inherently lossy. Even if a vendor is receiving the data directly from the venue just 1 hop away over a cross-connect, your vendor can lose data and have gaps that aren't present in another vendor's data. In one reputable investment bank I know of, they have these legacy switches and low-spec Intel NICs collecting the raw feed and they lose about 10^-6 packets - the same could happen for a vendor.

Likewise, most venues have a distributed design to their market data gateways in the data center, where data loss could happen to a subset of instrument channels for a subset of data subscribers, and the only way to mitigate this risk is to have redundant cross-connects to separate gateways - but most vendors will cut cost here because the additional connectivity fee could be over ten thousand per month.

There's also backup and recovery mechanisms that will differ between vendors. Depending on how fast a vendor's systems recover and cutover to secondary gateways, you could also see different data.

Some modern venues implement their data feeds over TCP transport, which means your connection is stateful and you might have to re-initiate connection when the session restarts, and upon subscription, you get a dump of current state. One vendor's reconnection mechanism could be different from another's, causing them to have "later data" for a significant interval after the session restart.

## Parser errors

This is less likely to present itself in how the vendor parses the standard fields (e.g. price, time) or message types but more likely to show up in how the vendor parses the less common fields or message types (e.g. reference data or instrument definition messages, consolidated summary messages, matching engine status messages etc.)

For example, some vendors might provide "tick data" with augmented BBO that does not take into account the matching engine status, and during auction periods (e.g. those found in Tokyo or SIX Swiss), you could see inverted BBO.

## Lossy normalization

> "[...] there were errors which it was the job of the vendor to filter out"

> "If the vendor is filtering any records, avoid the vendor."

I hesitate to use the term "unfiltered data", because there's no formal definition of "unfiltered" - it is purely an advertising construct. There's either raw or normalized data. Most people consume normalized data from a vendor because it is more convenient than being a direct subscriber from a venue.

As soon as your vendor normalizes the data across more than 1 venue, they have to filter something out, because by definition, they're trying to find common ground between two different wire formats. It's just a matter of whether that something manifests in your typical workflow or not. This ties to your next two questions:

> do different aggregation methods result in (statistically significant) different levels of accuracy?

> Would this accuracy difference get washed out if you aggregated to e.g. the nearest microsecond?

No, it will not get washed out. Here's a corner case where a vendor might materially affect you with lossy normalization even if they are providing you every sequence number.

On some venues, the trade print is published separately from the order deletion associated with the trade. In practice, the order deletion could be say, 100s of microseconds after the trade print. On others, the trade print and order deletion associated with the trade is defined as one atomic event.

A vendor that provides an aggregated snapshot of the book, such as best bid and offer, from both venues has to decide which convention they're going to follow. Are they going to show that the BBO depth changed at the time of the trade, or at the time of the book update?

This is usually worse when a vendor renormalizes the data because they lack the technical capability. i.e. Rather than source and parse the data directly, they white label the data from another provider's (such as Exegy, OnixS, Redline, MayStreet, Vela, Activ) feed parsers or normalized data (such as Bloomberg B-PIPE or Refinitiv Tick History).

## Timestamping issues

Some vendors will discard all of the native timestamps provided by the raw feed and embed their own. Some will take keep only 1 out of two or more timestamps provided by the raw feed.

Another naive matter that you'll find vendors deviate on is how leap second corrections are handled.

P.S.: A shoutout to our project at Databento, where - for full disclosure - I am one of the engineers. As you can see from all of the above, there's no standard way to resolve all of the potential data quality issues. Instead, how we think of these implementation details and normalization is a more holistic concept: that the data needs to be at a quality suitable for passive simulation. i.e. You should be able to backtest a strategy that places passive orders and get simulated fills and event ordering that is accurate enough for productionizing your strategy.

Some tricks we use to achieve this that you might also find applicable at various data vendors:

- Full CPU offload of data capture to FPGAs to mitigate non-determinism in timestamping and packet loss.

- PTP time synchronization for sub-microsecond accuracy of timestamps.

- Writing our own feed parsers so we have transparency of any lossy normalization.

- Subscribing directly to raw multicast feeds from the markets and managing our own colos to mitigate errors passed on from third party providers.