Survivorship Bias and Missing Data in Equity Backtests
Summary
The document considers an equity backtest with fundamental and macroeconomic filters when some historical data for delisted stocks cannot be recovered. Its initial rule allows trades when a filter passes or its value is missing. The author worries that this may admit delisted companies too freely, then proposes allowing missing data only for firms with a specified bankruptcy filing and a delisting close to the trade end date.
The examples show why delisting labels need careful interpretation: a ticker change can look like a delisting even when the business continues, while a company that fails years later may have been healthy during the tested trade. These cases do not establish that the proposed rule removes survivorship bias. Missingness can still be systematic, and using a later delisting reason or date may introduce look-ahead information. The document raises the need to represent historical security identity, delisting outcomes, and point-in-time data coverage consistently, but it supplies no validation results or definitive solution.
Key ideas
- Missing historical indicators for delisted firms can alter which trades enter a backtest.
- Treating every missing filter value as a pass may create unrealistic selection behavior.
- Ticker changes and eventual bankruptcy do not by themselves show that a security was unavailable during a past trade.
- A rule based on delisting reason and timing requires scrutiny for look-ahead bias and systematic missingness.
- Plausible sample trades are not sufficient evidence that a backtest is free of survivorship bias.
Tags
Full text
# Is this a valid shortcut for backtesting free of survivorship bias? # Is this a valid shortcut for backtesting free of survivorship bias? We backtest a very complex equity strategy that uses dozens of different fundamental and macroeconomic indicators. To make this backtest free of survivorship bias, we first collected all stock tickers that were delisted during our test period. Then we tried to collect a complete set of all indicators for the delisted tickers via our market data providers. Unfortunately, for some indicators and delisted tickers, there is no valid data available from any of our providers. We have therefore implemented an kind of shortcut according to: DO TRADE IF condition(indicator)=True OR indicator IS NULL However, this approach is very conservative, as we know from the listed instruments that all conditions have a strong positive effect on risk and return. We have therefore carried out a deep dive of the trades on a sample basis and have frequently identified the following situation, for example: - There is a trade of Facebook in 2019. As the ticker symbol changed from "FB" to "META" in 2023, this trade is flagged as delisted. But obviously this has nothing to do with survivorship bias, as Facebook or META still exists and is doing good business. - There are often trades by companies that later became insolvent. However, the insolvency only occurred years after the end date of the trade and at that time the companies in question were doing good business and the returns sometimes were even positive. We have therefore adapted our logic somewhat to: DO TRADE IF condition(indicator)=True OR (indicator IS NULL AND delisting_reason=Form 8-K Item 1.03 filing AND trade_end_date-delisting_date<=1 year) With this logic, the backtest results look plausible to us. However, we fear that we have forgotten something here or that we are misinterpreting the definition of survivorship bias. Could anyone say whether our approach is valid and gives us realistic results?
Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.