Choosing Storage for Tick Data Used in Backtesting

# What's the most efficient way to store options and time series data for backtesting?


# What's the most efficient way to store options and time series data for backtesting?












I would like to know what database would you guys use for storing around 500GB-1TB of options and time series tick data.

The idea is to use it for backtesting so it would have to be as efficient as possible.

I'm also interested in what service would you guys use for storing it online so it's accessible to more people.

## Answer by databento (score 17, accepted)

https://quant.stackexchange.com/a/65575

If the only purpose is to backtest with the data, the primary (in some cases, only) access pattern is to seek to a start time and read all of the data serially through to an end time. Then, there is a strong argument for storing it in plain, flat files with binary encoding, i.e. dumping the data structs in their in-memory layout straight to disk.

Storing it with a DBMS like kdb or Vertica is quite wasteful because (i) you don't need any of the querying functionality, (ii) there is a substantial learning and maintenance curve, (iii) licensing can be expensive, (iv) there is less potential for optimization.

If you have an occasional use case for exploring the data with a REPL or scripting language, it could make sense to use a binary disk format with some structure, such as HDF5 or Parquet.

> I'm also interested in what service would you guys use for storing it online so it's accessible to more people.

If you want it to be accessible to others over WAN, this is mainly an independent issue of infrastructure and network setup. You could host it on your own machines and open up your firewall, but that requires some expertise and initial capital outlay. Moreover, it takes significant work to optimize your storage to support large numbers of concurrent reads and to scale your storage past single host, 2-4U physical form factor.

Using a cloud provider is a good option here because it reduces some of the complexity of managing the network and storage yourself and much of the initial capital outlay. For example, you could use AWS S3 and store those files with key names quite similar to how you would store them on local POSIX storage. This makes it easy for users to access the data over HTTP/S3 API. You could also use EFS if you need POSIX semantics or memory mapping.

Alternatively, if you still want to self-host because you have a very small footprint (say, <30 TB and <10 backtesting hosts) and only need internal users to access the same data over VPN or LAN, one good option is to set up a Btrfs or ZFS filer. You can pay a small premium to get one ready-built from Synology or FreeNAS and you can expose the data to your internal users over a VPN service like Tailscale.