Handling Missing Market Data in Trading Backtests

# Backtesting, how missing data points should be handled?


# Backtesting, how missing data points should be handled?












Assume a daily trading strategy. Obviously, there are weekends and holidays when the market is closed. Should those days be excluded completely or should the price be interpolated in some way to fill the missing data points?

## Answer by madilyn (score 1, accepted)

https://quant.stackexchange.com/a/37536

The key question to ask is: Can you act on it if you were trading live?

Both Antoine and Norgate's responses are correct in that respect. You can't trade on holidays, nor can you trade on an interpolated price. Both are imaginary events that you can't experience if you were actually live. Similarly:

- IBM stock stopped trading when U.S.'s antitrust inquiry was announced. After the fact, most "clean" data sets will have no data for IBM on that day, but you actually need to assume you were planning to trade IBM on that day.

- Your data is unclean on a data of extremely high volatility (e.g. Flash Crash). You can't retroactively go back to clean the data as the corrupted data is actually what you'd have experienced live.

This same principle applies to another case that people find harder to understand: even if the data were available, you may have to ignore it. Here are a few examples:

- A sequence of trades in your data were later found to be reversed. If you were trading on that day, you would still see those trades print before the reversal was known, so you can still match your orders against those trades.

- A market that you're trading on runs 23-24h per day and has half-days on holidays. You have practical reasons you can't keep trading on during those times, so the data should be excluded even if you have it.

## Answer by Richard at NorgateData (score 1)

https://quant.stackexchange.com/a/37534

Each security has its own set of trading days on which the market is open for trading. For stocks, typically this excludes weekends (Note: NYSE traded on Saturdays until Sep 1952) and also excludes designated public holidays. There are some extraordinary events that also affect weather the market is open - in recent times these are weather or terrorist-related.

Individual stocks may also have trading halts/suspensions applied to them, which can last several days in some cases.

A good way of normalizing your date series is to use the dates from a known good index, such as the Dow Jones Industrial Average (for US stocks).

Beware of futures trading because the holiday schedule is different for each futures contract. The exchanges publish a detailed set of session closes for each market. Futures can also exhibit limit up/limit down moves where trading is halted for the rest of the trading session.

Spot forex typically trades all weekdays except for Christmas Day and New Years Day. Some new exotic exchanges (such as Bitcoin) trade every day.

Since the markets are closed for trading, interpolation is not applicable since you cannot trade anyway.

If you are backtesting trading strategies with multiple legs or use calculations across multiple securities/indexes, you should reconsider whether any signals are applicable in these scenarios.

## Answer by user1130176 (score 1)

https://quant.stackexchange.com/a/39462

For backtesting, through interpolation, take the mean of day before and day after. But note, you may want to flag that security as having interpolated data. If you're missing more than one day, just assume linear and interpolate accordingly. As others have stated, this is super risky because it may lull you into a false sense of security, but for back testing, if you have 10s of thousands of security-day tuples, and < 1% need interpolation, you can use this technique for global conclusions, but not for specific security or day conclusions.