Multiple Strategy Testing: Reality Check, FDR, and Out-of-Sample Validation
Summary
The document compares White’s Reality Check with the Benjamini-Hochberg-Yekutieli procedure for screening a large set of backtested strategies against a benchmark. It describes the Reality Check as accounting for dependence among test statistics through resampling, while noting that estimating dependence across many tests can be difficult when the available sample is limited. The answer says the standard BH procedure relies on assumptions about null p-values and that resampling extensions for correlated tests face similar dependence challenges.
It suggests Efron’s false discovery rate methods as a possible alternative for dependent statistics, but offers no comparative results or detailed implementation guidance. The discussion is brief and based on a response rather than a worked analysis. It closes by emphasizing evaluation of the selected strategy on data not used to choose it, an essential safeguard against selection bias that does not itself resolve all multiple-testing or regime-change concerns.
Key ideas
- White’s Reality Check uses resampling to account for dependence among strategy test statistics.
- Estimating dependence across a large strategy set may be unreliable when the sample is small.
- The standard Benjamini-Hochberg procedure relies on assumptions about p-values under the null.
- Resampling extensions for correlated p-values inherit dependence-estimation challenges.
- The answer proposes Efron’s false discovery rate methods and recommends out-of-sample evaluation.
Tags
Full text
# White's Reality Check versus Benjamini-Hochberg-Yekutielie Procedure # White's Reality Check versus Benjamini-Hochberg-Yekutielie Procedure I'm backtesting about 1k different strategies / permutations of strategies and I want to identify which if any of the strategies are better than the benchmark. Based on my readings, I feel like I've narrowed it down to White's Reality Check and the Benjamini-Hochberg-Yekutielie Procedure (with c(M) set to 1). What are the pros/cons of the two approaches? Is one decidedly superior? Is there a better approach besides these two? Thnx ## Answer by James (score 1) https://quant.stackexchange.com/a/15041 White's approach estimates the dependence structure (simply speaking, the variance-covariance matrix) of the test statistics (such as t-values or p-values) that are used for deciding what strategy is superior. The estimation of the dependence structure is not explicit, but that's what happens during bootstrap or simulation. Typically, the amount of data is not enough to estimate it (think of how much data is required to estimate 1000*1000 variance-covariance matrix reliably) but as far as I know that problem is swept under the carpet in all of the papers of White and his followers. BH (1995) assumes that the p-values are iid U(0, 1) under the null hypothesis, with a couple of technical caveats that are not going to be of help to you. Then, they also followed the idea of White, i.e. tried working with dependent p-values via resampling in this paper > Yekutieli, D., Benjamini, Y., 1999. Resampling-based false discovery rate controlling multiple test procedures for correlated test statistics. Journal of Statistical planning and inference, 1999 but the limitations of this approach are exactly the same as those of White. I would suggest using Efron's Fdr/fdr because, in a certain sense, it provides a viable solution when the test statistics are dependent, among other things. You can start with this paper, and the corresponding book can be found on Amazon. The locfdr package is no longer available in R, but I just downloaded and installed it separately. All of that being said, the ultimate test is seeing how the "best" strategy works out of sample.
Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.