Why Backtest Performance Ratios May Not Predict Future Strategy Returns

# Which metric is most predictive: Mean, Sharpe, Calmar, ...?


# Which metric is most predictive: Mean, Sharpe, Calmar, ...?












Suppose you have created a new trading algorithm:

- by varying the params of the algorithm, you get a large number of similar trading strategies (e.g. slightly different trigger thresholds, stop loss thresholds, etc.)

- for each strategy you can calculate the historical daily returns and various historical performance metrics

- now, you must pick 1 single strategy that will be traded in the market tomorrow (i.e. out-of-sample)

What performance metric will help you select the optimal strategy?

I did some simulations (details below) and for example:

- I found that ${historical \space average \space returns}$ are a terrible predictor of future performance

- while ${sharpe}=\frac {daily \space return \space mean}{daily \space return \space dev}$ is the best predictor I found so far

However, I have no idea why it is so and if there are even better predictors. Help!

## Answer by develarist (score 1)

https://quant.stackexchange.com/a/54442

The sortino ratio is also important for evaluating trading strategies. also the omega ratio.

The question is a poor one though. Each of the mentioned ratios will be the most predictive at predicting ... THEMSELVES! respectively. you don't use the calmar to predict the sharpe. ok, what you are probably actually asking is which of the performance metrics suffers the lowest estimation error. that takes time to understand what exactly each is calculating by breaking up the ratios into their components while going through the hundreds of existing studies that already report their reliability during estimation/prediction. the learning speed of each is of no interest either since none of the metrics listed are computationally intensive. but even after all this: why would any of them give you an edge, in isolation or in combination, in the first place? They are merely post-evaluation tools intended for analyzing realized performance, but unfortunately sold off by fund managers as decision criterions. Two very different purposes.

Due to the estimation error each possesses, and their mis-useage for prediction rather than post-evaluation that they are intended for, it should be clear why none of the performance metrics, or even any sort of combination of them, will ever achieve the ideal "equity curve".

Then bringing in the idea to adaptively change the trading strategy's optimality criterion (sharpe on day 1, calmar on day 2 and sortino on day 3) requires a step in between: a criteria in the middle that identifies the conditions under which a certain one of the performance metrics is selected. That step has not been considered in the write-up so far. Not that it will get you anywhere, but it will turn the whole point of the strategy on its head because the strategy is basically one that does not stick to a fixed game plan but executes based on a much longer decision process that is probably aimless in theory. On day 1 you want the best risk-adjusted performance, and then day 2 you want to control for drawdown? How would that sound for an investment policy? Investors usually just want one thing: high cumulative returns, the more money the better, without exception. After all, thinking that the selection of a certain performance metric for one specific trading interval will consistently make you richer? think again.

The biggest mistake in designing trading strategies is to believe that having a whole toolkit of well-known magical formulas and throwing them under the bus will make a bunch of profit. That's not what finance is about. Step back and draw up a theory as to why what you are doing will accomplish a certain result (consistently, and for a variety of assets that have different statistical characteristics) rather than just plugging and playing whatever you see in a finance textbook.

with all this said,what is the formula there for that "out of sample equity curve"? also, the ideal one doesn't look curved.