Correcting for Multiple Testing in Trading Rule Optimization

# Multiple comparison problems


# Multiple comparison problems












I recently read a blog entry where some statistics were generated for a common technical analysis indicator. Below is the link. My question shows up close to the bottom under the name bill_080, in the "Response" section:

http://www.portfolioprobe.com/2011/04/16/a-bit-of-analysis-of-the-dow-golden-cross/

My question here is the same question as in the link:

```
 ".....In the paper, they were able to come up with a “Universe of Calendar Effects”
 of 9,452 rules (and a reduced set of 244 rules) to generate the statistics.
 In the “Golden Cross” scheme, can you think of a way to put a number to the
 “Universe of Golden Cross Effects”, and then look at the various p-value,
 t-stat, or whatever stats apply?....."
```

I have been dealing with similar problems for a while, and I do have a very CRUDE way to deal with this issue. However, I'm looking for any suggestions from others.

Edit 1 =========================================

To be more specific, the "Calendar Effects" paper that I referenced in that link discussed the multiple comparison problem where the optimization was over a DISCRETE range. As a result, they were able to count up the "Universe of Calendar Effects" to be 9452 rules. So, they were then able to correct any p-value/t-stat/etc statistics to reflect this universe of possibilities.

If I try to apply this same technique to a problem that has a CONTINUOUS range or ranges, as in the "Golden Cross" case, the first n-day moving average (which was optimized over a continuous range, giving a 200-day moving average) and the second n-day moving average (which was optimized over a continuous range, giving a 50-day moving average), how do you comfortably nail down a discrete number (the "Universe") to correct the p-value/t-stat/etc for this CONTINUOUS problem?

My production/inventory problem is essentially the same as that "Golden Cross" problem. I have several variables (it's my choice how many) that can be bumped until I hit the optimum combination, but when I do that, how do I come up with a discrete number to correct for the multiple comparisons that I just carried out? To make things easy, let's just say that the required correction is the "Sidak Correction" given by:

http://en.wikipedia.org/wiki/Multiple_comparisons

```
alpha1 = 1 - ((1 - alpha2)^(1/n))
```

Let's say that if this was a normal one-time, no optimization situation (n=1), I might use alpha2 = 0.05 (5%). What is the equivalent of this 5% value if I optimize 1 variable, 2 variables, 3 variables, etc. If I add, let's say, 100 to n for each optimized variable, I'll get the following correction table:

```
Number of Optim Var     n    alpha1
                  0     1   0.05000
                  1   101   0.00051
                  2   201   0.00026
                  3   301   0.00017
```

So, if 100 is the "right" adder, then that "Golden Cross" problem would have to match alpha1=0.026% (0.00026) to be consistent with an alpha2 of 5%. Just guessing, but I doubt the Golden Cross scheme would pass the test.

Similarly, if I optimize my inventory problem using 3 variables, my target alpha1 is 0.017% (0.00017). That is a tough hurdle to clear. At an adder of 100, one optimization variable clears the hurdle, but no more.

So, is ADDING 100 per variable too much, not enough? Should it be exponential instead of an adder, so that the above table has n=1, 101, 10001, 1000001? Some other scheme? That's what I'm trying to justify. Has anyone done a study or seen a paper on correcting p-values/t-stats/etc for CONTINUOUS variables?

## Answer by Dirk Eddelbuettel (score 6, accepted)

https://quant.stackexchange.com/a/1008

David Aronson spent 500+ pages detailing this one key idea of multiple comparison problems in searching for trading rules---so have a look at his book called 'Evidence-Based Technical Analysis.