Survivorship Bias and Statistical Power in Active Manager Evaluation

Summary

The document considers whether conventional significance tests, including a t-test, are appropriate for judging active managers when test power is limited by noisy returns. Its response emphasizes survivorship bias: managers with observable track records may be a selected group because poor performance can lead to withdrawals and business failure. Testing only those who remain can make evidence of skill appear stronger than it is, even if managers’ returns were generated without skill.

The answer also points out that available performance histories are often monthly. It recommends estimating the test’s statistical power when working with such data, rather than interpreting significance thresholds in isolation. The document does not provide a power calculation, dataset, or comparison of specific tests, and it does not prescribe a lower significance bar. Its central lesson is that selection effects and the amount and frequency of data constrain inference about manager skill.

Key ideas

• Survivorship bias can make observed manager track records look unusually strong.
• Managers who fail may disappear from the sample used for significance tests.
• A t-test on surviving track records can overstate evidence of skill.
• Monthly observations may limit the ability to detect skill.
• Estimate statistical power when evaluating significance with limited data.

Tags

# Statistical Power and Active Management


# Statistical Power and Active Management












I was reading an article recently that pointed out the dangers of using statistical inference in evaluating active managers as the power of statistical tests diminishes with the variance of the underlying data. I can understand the case for requiring more stringent tests like White's reality check, but could you ever make the case that tests for statistical significance should have a lower bar or that a t-test is not useful in evaluating active managers?

## Answer by Jase (score 3)

https://quant.stackexchange.com/a/8580

The main problem with hypothesis testing is going to be survivorship bias. Any manager with a track record you're looking at is only there because they haven't performed badly -- if they perform badly then investors withdraw their money, they collapse, and you don't have their data to do the hypothesis test on! So even if all the managers were investing in geometric Brownian motions then survivorship bias will make a lot of managers for which there is sufficient data pass the skill hypothesis test with flying colours.

The other issue is that most of the data you'll be able to get is monthly. You need to do a power estimation of your test if you're using monthly data.

Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)

This summary was written by Stratmill's research agent from the original; it is not a copy of the source.