Statistical Validation, Backtest Overfitting, and Trading Research

# Encyclopedia of Statistical Tests


# Encyclopedia of Statistical Tests












I am aware of: Encyclopedia of Chart Patterns, Encyclopedia of Technical Analysis.

### Question

I'm wondering if there's something similar, but in the form of:

"Encyclopedia of Statistical Backtesting to see if you're being fooled by randomness" and "Encyclopedia of techniques for generating random forex patterns."

## Answer by abstract (score 2)

https://quant.stackexchange.com/a/7444

The best book concerning statistical validation of objective technical indicators would be David Aronson's highly acclaimed Evidence-Based Technical Analysis.

While the first quarter of the book is spent establishing the distinction between chart eye-ballers and objective verifiable TA, the latter portion of the book is an absolute goldmine as it starts off with the basics of statistical inference and then covers the monte-carlo permutation method and bootstrap tests extensively. In the end, numerous technical indicators were tested on the S&P and none of them exhibited statistically significant predictive power, although the author did mention that combinations of such indicators could be lucrative.

## Answer by SRKX (score 1)

https://quant.stackexchange.com/a/4194

I believe this is very difficult to do because of the different nature of statistical tests. Some of them are used to test the assumption of normality, some of them allow you to compare the volatility of different samples, some of them allow you to determine the suitability of a specific model.

Essentially you will find the basic ones on any good statistical book with a hypothesis testing chapter. For the more advanced one, you will need an introduction to the topic first which require ... a book by itself.

## Answer by DangerMouse (score 1)

https://quant.stackexchange.com/a/4200

You probably would do well to understand the question you are asking. Read the seminal text "Subset Selection In Regression" on the subject.

An encyclopedia of statistical tests applied without understanding would be as damaging to your wealth as both the Encyclopedias you mention above - if I read into your question correctly.

## Answer by user7056 (score 0)

https://quant.stackexchange.com/a/4195

There is "A Review of Backtesting and Backtesting Procedures" by Sean D. Campbell.

It is related to VaR backtesting, but the article discusses the general properties of unconditional coverage and independence, giving the statistical power properties of different backtests.

I do hope it helps.

## Answer by bill_080 (score 0)

https://quant.stackexchange.com/a/4198

If you run enough tests, you are guaranteed to find something that "works" in a backtest. The problem is.....Does it work in real time trading? If not, then you were "fooled by the backtest". If it does work, the next question is.....For how long? Markets evolve 100% of the time. So, when the thing that "works" eventually dies out, was it really there or were you just lucky?

The bottom line is.....Is it reasonable to assume that what you're looking for is predictable? If not, then why waste your time. If so, then what advantage do you have over others in exploiting that predictability?

## Answer by Vazgen (score 0)

https://quant.stackexchange.com/a/4236

I'm also looking for something like this, a book with setups and examples of statistical tests specifically for trading strategies. I picked up "Probability and Statistics for Finance" (Frank Fabozzi Series). I'm still reading it but I have not yet encountered an example of an application to an actual trading strategy/backtest. However, it's relevance and examples in general finance does help give it a context for the statistics/probability concepts described.