Why Backtest Significance Requires a Defined Null Model

# How to estimate the probability of Clustering illusion in our backtest result?


# How to estimate the probability of Clustering illusion in our backtest result?












Suppose I have a strategy, I run a backtest on it in only one symbol (suppose the historic data to backtest is 25000 candles). The results of that backtest is:

Total Trades = 50 TakeProfit/StopLoss ratio = 2:1 for every trade Avg TakeProfit % = 3% (that means avg StopLoss % is 1.5%) Win Trades = 30 (60%) Long Trades = 20 (40%)

How can we test if those results were just "luck" or we discover a pattern? (at least in our tested data)

I was thinking in something like this: Run a random strategy, with similar deviation to our strategy that is (for every candle): Probability to open position in current candle: 0.002 (50/25000) Probability to go Long: 0.4 (40%) TakeProfit %: 3% Stop Loss %: 1.5%

And I run that random strategy using the same data. One time, two times, over and over until I find a result similar to my strategy (near 50 trades, and near 30 win trades). And then I count how many random strategies I had to run to find those results. Suppose I executed 100 random strategies and only 1 of those has a similar result. Does that mean the chances of a false positive in my test is 1%?

// not sure if I'm explaining it well, sorry for my english :/



## Answer by Dave Harris (score 3)

https://quant.stackexchange.com/a/48790

False positives and false negatives are concepts from Frequentist statistical inference. They depend on people not data mining. Using Keynesian notation, they test $\Pr(X|\theta)$. That is to say, they test the probability of observing a result as extreme or more extreme than the sample given that the model is true. In more common notation they test $P_\theta(X)$. You are not doing that in what you describe.

You are not stating a true model. You are attempting to pass through all possible models combinatorically. You cannot test for false positives using that methodology.

To understand why, you should understand how you would test for a false positive. Let us pick an elementary example, such as how often the sample mean will produce an extreme result. If you do not have calculus or analytic tools available, as you would not in what you are describing, then you would create tens of thousands of samples of size N.

You would construct the implicit density function for that simulation. You would find the $\alpha$ cutoff you would use. Using that, you would be able to approximately control your level of false positives by making that your $\alpha$ cutoff.

In other words, you cannot use actual data. Frequentist statistic work in the sample space. In your case, that would be over the set of all possible trades that could ever exist over your time frame.

The only way to estimate false positives and negatives is to have a single theoretical model to work as your null.

What you are describing is actually why backtesting as a way to find a strategy cannot work. There exist a countably infinite possible number of possible models. If you specified a single security's price as a difference equation, you would note quickly that it is a strictly path-dependent model. By backtesting to get a strategy, you are confusing noise for signal.

One last note, for an inferential strategy to work using a Frequentist method, your model has to have a correct specification. Since you do not know the specification, you cannot succeed. If your concern is inference, then use logic alone.

Construct a logical specification. If you can do that, then you can control for false positives and false negatives by controlling for statistical power.

EDIT

Let us start with the assumption that you have a fifty percent chance of having a winning trade. That assumption is only true if you randomize two things. First, you have to randomize the security and transaction time. Second, you have to randomly decide whether to go long or short. Your assumption that fifty percent of all trades result in a win isn't supportable.

If your concern is "can I believe my results," then the answer is probably "no." You can know that without a statistical test.

First, there is a problem with backtesting. You are assuming you were the person making the trade. However, if you had actually been in that period of time then you would have had to outbid that transaction to make it onto the tape. The time series is strictly path-dependent. You entering the path changes it forever. In order for your trades to happen, you would have had to shift the supply or demand curves. If you are involving any real amount of money, that shift could be very substantial. Even WalMart's stock can shift noticeably from a block trade. So if you backtest, you should be shifting your entry into the record by an estimated shift in the supply and demand curves.

Second, the trades on the ticker are not necessarily in the order in which they actually happened. Large orders trade "off the tape." They are inserted later when they close at a point that the transaction will not shift the market. If your strategy depends in any way on the order in which transactions happened rather than the general level, then you can chuck your strategy into the dustbin. Large orders are reported as weighted averages and not actual values. A trade of 101 and 103 for 500 shares each is reported as 102. If 102 happens to be greater than any other trade, it will be reported as the daily high even though 103 happened. U.S. transaction data doesn't include all real prices or times.

Third, the distribution of returns for going concerns is a truncated Cauchy distribution. That will defeat any attempt to think of the problem as a binomial as you will have way too many runs due to random chance alone relative to something such as the S&P 500.

Use logic as your protective tool. Does it make sense independent of if it works? Take your strategies, before you see your outcomes, and ask yourself does this strategy make sense. If it does, then test it.

Your assumption of a fair coin doesn't work for this type of problem. You are working with a problem where the first central moment does not exist. You are trying to use math appropriate for mesokurtic or platykurtic data when your data is leptokurtic.

While it would be valid to perform transformations of the data to cause it to be better behaved, that requires a substantial amount of skill to do correctly. It still will not solve, "did I discover something?"

You could possibly use Bayesian non-parametric methods but they would require years of study as they work in an infinite dimension model space. They are the closest thing to what you are trying to do.

## Answer by Alex C (score 1)

https://quant.stackexchange.com/a/48801

I think what you have in mind is similar to the Cowles Test, described by Alfred Cowles in his 1933 paper Can Stock Market Forecasters Forecast? (link)

Cowles wanted to evaluate the trading performance of an advisor who tells people when to be in the stock market and when to get out. He compared the actual performance to random switching in and out of the market.

> For example, to compile a purely chance record to compare with the actual record of a forecaster whose operations covered 230 weeks from January 1, 1928, to June 1, 1932, we first determined the average number of changes of advice for such a period, which was 33. Cards numbered from 1 to 229 were shuffled, drawn, re- shuffled, drawn, in all 33 times. Thus 33 random dates were selected as of which forecasts were to be changed. [Page 318]

The performance of the randomly switching strategy was evaluated, and the process repeated. Because of computing limitations at the time Cowles only generated 24 random sequences of switching dates, but today we can easily generate 1000 monte carlo sequences.

By finding where the actual performance ranks along the random performances, Cowles was able to evaluate the statistical significance of the results. If the human strategy is in the top 5% of random strategies, for example, we can say that this performance is significant at the 5% level.