Timestamp Ordering and Cross-Feed Skew in Market Data
Summary
The document discusses timestamp reliability in free historical data from Gain Capital, especially for combining non-FX instruments or multiple feeds in a backtest. One respondent reports that observations were correctly ordered within individual CSV files but that the files themselves appeared out of order, leading them to use another data source.
The discussion also cautions that timestamps from separate providers may not represent simultaneous arrivals. Supply-chain intermediaries, network delays, and processing latency can create skew between feeds, and some vendors smooth or aggregate data over intervals to align sources. A later response says another service’s data was tidied and ordered for use. These are practitioner observations rather than a formal audit, and the document gives no quantitative measure of the timestamp errors or verification method. It highlights that apparent synchronization should be checked before using multi-asset or cross-market data in timing-sensitive analysis.
Key ideas
- Individual data files may be ordered even when a collection of files is not.
- Combining feeds can introduce timestamp skew from provider chains and network or processing latency.
- Matching timestamps across instruments does not prove that events arrived simultaneously.
- Aggregating data over time intervals can conceal finer timing differences.
- The source offers practitioner reports but no measured error bounds or systematic timestamp audit.
Tags
Full text
# Quality of GAINDATA timestamps # Quality of GAINDATA timestamps Does anyone have a view on the quality of the timestamping of GAINCAPITAL's free historical data. There is non-FX data there and I wonder if the timestamps are in sync? ## Answer by thwd (score 2) https://quant.stackexchange.com/a/8026 I composed Gain-Capital's historical data files into one series for back-testing a while ago and too noticed that they are out of order. Each CSV file contains correctly ordered data points but it seems the CSV files are out of order (maybe wrongly named). I went for another source in the end. > If Index A comes in at 12:00:01.001 and Forex B comes in at 12:00:01.001 - did they in reality arrive at the same moment? I work at one of the largest market data providers there is. Out of experience I can asure you that there's is always a time-skew between two data-sources. This skew is due to mainly 2 factors: - Number of intermediary provider/consumer pairs in the data supply chain - Network and computation latency Even high-profile market data providers struggle with this and often the output-quality is not as good as you would expect. Some providers will try to align different data sources by aggregating them over a period of 15 minutes before output - this is called "delayed" data-quality. > Are you willing to share what you other source of data is? See this answer. ## Answer by Jared Broad (score 1) https://quant.stackexchange.com/a/8094 We gave up on using Gain's data in the end and partnered with FXCM to provide our FX data. We tidied and timeseries ordered it through our IDE. Disclaimer - I'm a founder at QuantConnect.com
Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.