Choosing Robust Backtest Parameters Across Performance Surfaces
Summary
The document considers how to choose two strategy parameters from an in-sample grid using Sharpe ratio, Sortino ratio, and maximum drawdown. Its central lesson is to assess the pattern across nearby parameter settings rather than automatically selecting the single best observed cell. A parameter combination that performs well amid a broader favorable region may be a more stable estimate than a sharp isolated peak.
The response focuses mainly on Sharpe, reasoning that it reflects the full return series while drawdown depends on one observed worst episode. It reads the supplied tables as favoring a region around six to nine months and twenty to thirty stocks, while noting that the exact choice may depend on strategy intuition and metric priorities. The discussion is qualitative and based on the provided in-sample results; it does not supply a formal selection rule or out-of-sample validation, and the suggested neighborhood should not be treated as proof of future performance.
Key ideas
- A broad region of strong results can indicate more robust parameter choices than an isolated in-sample maximum.
- Large performance changes between nearby settings may signal sensitivity to noise or a special sample circumstance.
- The response gives more weight to Sharpe because it summarizes the return series, whereas drawdown is based on one episode.
- The suggested parameter neighborhood is a judgment from the displayed grid, not a formal rule or out-of-sample result.
Tags
Full text
# Selection of optimal backtesting parameters
# Selection of optimal backtesting parameters
Suppose I backtest some strategy on in-sample data while varying two parameters, say $X$ and $Y$. $X$ can take the values $\{3,6,9,12,15,18\}$ while $Y$ can take $\{10,15,20,25,30\}$. I want to select appropriate values of $X$ and $Y$ for testing the strat on out-of-sample data. The tables of `sharpe` ratios, `sortino` ratios and max drawdown (`dd`) are as follows:
```
> sharpe
No_of_stocks X3.month X6.month X9.month X12.month X15.month X18.month
10 0.2923854 0.2485804 0.3116992 0.2356674 0.2711520 0.2535123
15 0.2801226 0.2757317 0.3362495 0.2420944 0.2369459 0.2293062
20 0.2887139 0.2953232 0.2952627 0.2979134 0.2553015 0.2249027
25 0.2736581 0.3268325 0.2971468 0.2896665 0.2401743 0.2240485
30 0.2761537 0.3423867 0.2964909 0.2905532 0.2948999 0.2137761
> sortino
No_of_stocks X3.month X6.month X9.month X12.month X15.month X18.month
10 0.4080662 0.3380257 0.4144185 0.3087768 0.3521293 0.3231242
15 0.4013694 0.3842653 0.4503256 0.3174395 0.3080369 0.3006281
20 0.4172279 0.4103027 0.3873160 0.3958244 0.3307235 0.2933315
25 0.3925792 0.4787884 0.3940304 0.3848995 0.3095552 0.2892468
30 0.3987750 0.4990707 0.3906656 0.3826982 0.3863327 0.2721931
> dd
No_of_stocks X3.month X6.month X9.month X12.month X15.month X18.month
10 0.5153225 0.5414108 0.4568199 0.5361848 0.5332630 0.6036963
15 0.4821441 0.4207504 0.3996013 0.5099167 0.5355697 0.5306460
20 0.4246441 0.3970251 0.4178547 0.3985710 0.4945658 0.5100034
25 0.4326678 0.2433439 0.3900689 0.4038422 0.5093099 0.4805518
30 0.3173227 0.2467464 0.3621063 0.3928437 0.3686759 0.4893400
```
As you can see, the 6-month;30-stock combination gives the best results in general. Someone told me that if, with a small variation in any one of the parameters, the value of the metric changes drastically, I should probably not select those combination of parameters (I guess it could be called an estimation error?). If we move from 6-month;30-stock to 9-month;30-stock, the sortino ratio changes pretty drastically from .499 to .39, and the max drawdown changes from .247 to .362.
So by that logic, I probably need to select the 3-month and 20-stock combination, because the metrics don't change a lot if we vary the no. of stocks or no. of months. My question is: is it okay to just blindly select the combination of parameters that give the best results (6-months,30-stocks) or should I also take the variation in metrics into account (3-months,20-stocks)?
Thanks in advance!
## Answer by GodLovesATrier (score 3, accepted)
https://quant.stackexchange.com/a/26371
Firstly I'm only concentrating on Sharpe, this is the most robust of your metrics. Consider the difference between sharpe and drawdown, sharpe contains a contribution from every return in your result, drawdown is just one observation of whatever happens to be the longest run of negative returns. Of course it's your choice if you want to place more weight on Sortino or drawdown.
Here's the data turned into a heat map, blue highest and red lowest.
This is more of an art than a science so I will attempt to explain what my reasoning would be:
- Roughly as you increase the number of months past 9 things get worse so all other things being equal I would go for something with a lower number of months.
- The lower left triangle of values looks better than the upper right.
- Although there is a local maxima around (15 stocks, 9 months) the other local maxima around (30 stocks, 6 months) looks better. It's also bang in the middle of this better triangle.
Remember you are trying to estimate from noisy observations what the best parameters are. If a small change in a parameter causes a large change in the observed result who knows what sort of "special" circumstance you have wandered into.
In conclusion I'd pick something between 6-9 months and 20-30 stocks. Hopefully you will have some intuition which will help you pick the exact amounts, namely that more stocks or lower periods should be better.Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.