Extracting Exchange Market-Data Payloads from PCAP Files
Summary
The document describes how to reach exchange market data carried inside packet captures when building a feed parser or order book. For UDP traffic, the application payload follows the Ethernet, IP, and UDP headers, which must be skipped to access the feed data. A packet inspection utility can still help verify that the packet structure is being interpreted correctly, even if it does not directly provide the desired payload.
It also presents a quicker extraction approach using a packet analysis command-line tool to print packet data as hex, which can then be converted into binary files. The example is intended for historical exchange samples and mentions that some sample captures contain payloads without the same outer framing. These are practical parsing suggestions rather than a full feed-handler implementation: packet formats, message schemas, sequencing, and exchange-specific protocol details still need to be handled separately. The document provides no performance comparison or validation results.
Key ideas
- Exchange feed data in UDP captures sits after the Ethernet, IP, and UDP headers.
- A parser must separate network headers from the application payload before decoding feed messages.
- Packet inspection tools can help confirm that packet structure is understood.
- Hex payload extraction followed by binary conversion offers a quick workflow for sample captures.
- The examples do not cover exchange message schemas or full order-book reconstruction.
Tags
Full text
# Decode stock market data from C++
# Decode stock market data from C++
As practice, I have been wanting to parse exchange data and try to build an order book algorithm on my own. I found some sample data from NYSE: ftp://ftp.nyse.com/Real%20Time%20Data%20Samples/NYSE%20XDP/.
I tried using tcpdump to read the pcap data. However, that does not get the underlying market data. Does anyone have suggestions on how I could parse the pcap file to read the market data? I am currently hoping to do this in C++ as most high frequency trading places use C++.
Any suggestions and recommendations will be very helpful. Thank you.
## Answer by chrisaycock (score 3, accepted)
https://quant.stackexchange.com/a/55098
I've created an example for how to access UDP packets in a pcap file.
The gist is that you have to skip the Ethernet / IP / UDP headers to reach the payload. That's what gets passed to your feed handler.
As for tcpdump, it won't pass the payload to you, but it's still helpful for verifying that you understand the contents when parsing. Eg.,
```
tcpdump -r NYSE_XDP_IMB_2.2.pcap -e
```
## Answer by databento (score 4)
https://quant.stackexchange.com/a/55102
I recommend @chrisaycock's answer for completeness. However if you want a quick and dirty way of extracting the payload, you'd use `tshark` instead of `tcpdump`:
```
tshark -r NYSE_XDP_IMB_2.2.pcap -T fields -e data
```
This can be useful sometimes because many exchanges (NASDAQ, Australia and SIX Swiss come to mind) typically send you historical samples with only the payload.
For instance if you want to transcode your directory full of NASDAQ pcaps into their sample binary format, parallelized across 40 cores, you just need a magic one-liner:
```
find . -maxdepth 1 -name "*.pcap" | xargs -I {} -P40 sh -c 'tshark -r {} -T fields -e data | cut -c 41- | xxd -r -p > `basename {} .pcap`.bin'
```Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.