Hardware Design Principles for Low-Latency HFT Systems

# Ultra-High Frequency Trading Help


# Ultra-High Frequency Trading Help












I am putting together an Ultra-High Frequency desk and need to answer the following questions for ordering some rack servers to process about 2 GB of data per second. If anyone has worked at a HFT desk before can you help me out? What kind of software do big HFT shops run?

- How many processors are required, and what clock speed?

- How much memory required?

- How much internal storage would you need?

- Do you require any external storage?

- How many network ports are required?

- How many fibre connections are required?

## Answer by Theodore (score 19)

https://quant.stackexchange.com/a/42094

This question has been re-opened again after (rightly) being closed as too broad for the purpose of clearing some misconceptions regarding one of the answers here.

The main idea that is to be stressed here is this: When it comes to high frequency trading, biggest or "as many as you can get" is rarely true.

Not only is it false, it is impossible in terms of space requirements and other factors that limit what HFT architecture can look like.

Let's assume you have all the resources, including the whole monetary aspect of this. Let's go over some of the inherent limitations for what co-located HFT machines look like.





Depending on the specific hardware implementation, let's go through a scenario investigating the basic structure of what that would look like.

For this example, say you can have all different tasks running on separate threads (talking specifically about HFT here) doing whatever they may be doing for the application (e.g., a thread for dealing with the market data coming in (a feed handler), one for processing the market data, and the exchange / order gateway thread). Let's discuss in more detail what this three processes entail for this specific hardware setup.

Note: In no way am I saying the following setup is industry standard nor that it is even a smart one. It is just an example demonstrating some of what the strategy might be doing with respect to hardware.



- Signal Discovery / Generation: The job of this thread should be fairly self-explanatory, it is to take in the market data and based on what the basis of the strategy is, it will make some decision based on what is going on. Something important to note here is that often this is the part that you see NOT in assembly / written at the hardware level given that computations of this nature are dangerous and taxing to be iterating at the hardware level.



But why is this the case? Why do some parts of this have to be dealt with in a sequential manner? Mainly, due to the fact that, the results of what each thread is doing must be passed along onto the next thread, this is easy to visualize with our rudimentary example of what each thread is doing in our fake HFT application.

What might be bad about this?

Well for one, you just don't need all these threads!

So, from this answer:

> Q: How many Processors are required, and what clock speed? A: as many as you can get

doesn't make much sense at all!

For the most latency-sensitive aspects of a HFT system, you are going to see something that looks more like this:

Single threads isolated to single cores. Why? There are two main reasons I can think of: the OS does not get in your way (i.e., if everything is locked to one core and one thread, it is easier to deal with the OS and prevent it from delegating any other tasks to that thread, which saves you precious cache). Secondly, any implementation of lock-free programming is much easier in a setup like this. For a single thread operating on a single core, you can have single-producer single-consumer calls that integrate what happens on this core and ensures its efficient path over to the strategy core which operates in a similar manner.

Now, onto memory:

> How much memory required?

Not much, actually. Definitely not "A: as must as you can get"

Think about it this way: do you want to be reading from memory for doing HFT?

No, you don't and you won't. Given the deterministic nature of this activity, it is imperative that your memory path be quickly accessible and not susceptible to a miss and read from RAM, so keeping it low level in CPU cache is by far the best option for this.

Now, storage!

> How much internal storage would you need?

Again, similar to the deal with random access memory: reading from memory is bad, but reading from storage? This would be catastrophic. The data from the exchange is being stored, but not onsite! Why would you spend money on co-location space and take it up with storing tick data? The answer is that you wouldn't.

> How many network ports are required?

The other answer covers this for the most part, it depends on the exchange and how it operates however generally speaking you would not see more than two or three. Theoretically, you only really need connections to order handlers of the exchange and that is another example of an inherent limitation; you get whatever bandwidth the exchange supports. This relates to the last question about fiber and bandwidth: given that you reach a limit for the bandwidth "as many as you can get" is silly and nonsensical as was stated prior. You will reach a bandwidth limit and more fiber does absolutely nothing for you.

The OS that runs on the machines depends on the hardware, and mostly that aspect of the system is less of an area of concern. Just something light on the resources that makes kernel networking easier. If you must know of a specific operating system that is used on HFT machines, I know of a successful firm using CentOS, which is a derivative of the upstream Red Hat Enterprise Linux (RHEL). The latter runs traditionally on x86, x86-64 and Itanium servers.

If this had to be summed up into something extremely short, here it is:

- More CPU cores `!=` better HFT system. Isolating single key processes to cores makes more sense as the communication is easier (and quicker), and you eliminate potentially paying for what the quick path interconnect introduces into the system.

- the RAM of your system is largely an unimportant factor given that most of the tasks that are memory intensive should be done in CPU cache.

- Storage is an even smaller for a similar reason regarding RAM: data is not stored onsite and reading from storage would be terrible.



- More fiber `!=` higher bandwidth. The bandwidth is a factor again largely dependent on the exchange, more fiber will not do anything for you.

## Answer by wburzyns (score 7)

https://quant.stackexchange.com/a/786

Your question is very broad and cannot be answered by community as it requires a thorough analysis of your business needs. This isn't something that can be done while having a coffee break. Of course one may answer to your questions: "as many & as much & at least & the fastest" but your costs will skyrocket as there is virtually no limit... Do you really want to spend money on an F1 car while all your needs can be satisfied by a decent van?

Long story short: you'll do better by hiring somebody knowledgeable to do the job.

## Answer by glyphard (score 0)

https://quant.stackexchange.com/a/785

Q: How many Processors are required, and what clock speed? A: as many as you can get How much Memory required? A: as must as you can get How much internal storage would you need? A: as much and as fast as you can get, think 500GB SSD Do you require any external storage? A: yes, think in terabytes How many Network Ports are required? A: at least 2 How many Fibre connections are required? A: as many as you can get Which level of AIX software is required? A: the fastest version, with the fewest nonessential modules