Designing a Normalized Market Data Feed for Trading Systems

# Answer by madilyn (score 1, accepted)


# How to filter and normalize market data obtained from distinct sources (FIX 4.4, bloomberg, etc) in an algorithmic trading system?












I'm wondering if some of you known how to resolve this requirement:

I have to define the architecture of an algorithmic trading system (but I'm not an architect, so I'm trying to do my best). I have defined an initial architecture, but just now I'm stuck at the data feed handler component.

I mean, the system will receive market data from different sources (bloomberg, FIX4.4, etc) and must normalize that data to produce an usable data feed which it's supposed to be consumed by algorithms to make some calculations an create some orders, something like this:

mk data providers => data component => normalize data => usable data => algorithms (consume normalized data)

So, I'm wondering if you know a good way to make this or maybe you know a good opensource market data feed handler that can receive data from different providers and produce one clean and normalized stream of market data.

I will appreciate very much your answer. Thanks in advance.

PD: I have been doing some research and I found this:

- http://www.openmama.org/

- http://www.stuartreid.co.za/algorithmic-trading-system-architecture-post/

And for now I'm just reviewing those sites...

## Answer by madilyn (score 1, accepted)

https://quant.stackexchange.com/a/18679

If I understand your question correctly, you're asking what's a good design for a normalized feed. This is a somewhat trivial question of (i) picking which data fields (e.g. price, volume) to filter out from each feed and (ii) how to keep that in a trading system with minimal computational overhead.

### Regarding (i)

I highly recommend you approach this in a test-driven manner. In other words, figure out what data your application is going to use downstream and 'reverse out' what you need to normalize upstream.

A trivial example: If your strategy just needs prices, there's probably little use in normalizing the match numbers. If your compliance and post-trade analysis is keyed by transaction times rather than sequence numbers, then you probably don't need that either.

Two tips that I can give you are:

- Don't overstretch yourself. It's easy to make the mistake of trying to normalize too many from day 1. You don't want to waste time trying to get spot FX, swaps, equity options, equities, exchange-traded futures etc. all onto the same normalized feed. It will probably take you a very long time before you're dealing with all that, and by the time you have to, you probably have someone else redoing this for you anyway.

- Think a bit about how you will maintain depreciated fields and version changes you make to your normalized feed. No matter how exhaustive your initial design, I guarantee you will encounter changes over the course of implementation and real-time use.

### Regarding (ii)

Without knowing your exact architecture, the best answer we can give you is a collection of obvious software development tips: keep this in memory, minimize allocation and GC overhead, and minimize the number of times you're shuttling the data from thread to thread.

### Important note

If you're trading based on more than 1 data feed, this is not something you want to do alone. It's seriously not worth your time. There's a reason why vendor normalized data feeds are costly, they take a lot of time to maintain.

The part I don't get about your question is: Bloomberg is a normalized data feed provider whereas FIX is sometimes used by exchanges directly, why are you normalizing across the two? Then you're just introducing unnecessary overhead in the Bloomberg path. I would just use Bloomberg alone if that's the case.