Selecting Robust Betting Parameters from Backtest Results
Summary
The document considers how to choose among many Kelly-based betting strategies after testing a UFC fight prediction model. The model outputs win probabilities, and the user evaluates parameter combinations using bookmaker odds, return on investment, and maximum drawdown. The central concern is that selecting the highest-profit combination from one out-of-sample period may still overfit, while averaging parameters across many qualifying strategies may yield a poor combination. The main answer recommends examining the parameter space for regions where profitability changes little as parameters vary. A broad plateau of acceptable performance may be more robust than a narrow peak, since small parameter estimation errors are less likely to undermine it. Another response suggests regression over the parameter combinations, but the document provides no validation that this produces a reliable selection rule. The proposed robustness check is qualitative and based on a single stated evaluation period; further validation on independent data would be needed.
Key ideas
- Choosing the top-performing parameter set can overfit even when results come from an out-of-sample period.
- Look for broad parameter regions where profitability remains relatively stable as settings change.
- A strategy near the center of a robust region may tolerate parameter error better than one at a sharp peak.
- The regression suggestion is not supported by evidence in the document and would require separate validation.
Tags
Full text
# How to select optimal betting strategy from backtest? # How to select optimal betting strategy from backtest? I have written a model for predicting the winner of UFC fights. My model calculates the probability of each fighter to win a given match. I have back tested the model and found it to be very accurate, it predicts the winner around 65% of the time. The model is trained on 3 years of data and then tested out of sample on the past 9 months worth of data. I am trying to use my model's output (out of sample) and historic book maker odds to come up with an optimal betting strategy. This is based on the Kelly Criterion. I have 5 parameters that I think will affect the profitability of a betting strategy. They are based around fractions for kelly and handling uncertainty. What I did is write a program to create ~500k strategies with different weights for each parameter. I then run them through the past 9 months of data to determine their profitability. From those ~500k I can narrow down the strategies I'm interested in by filtering them by maximum draw down (20%) and minimum ROI (25%), this will bring down the strategies to around 20k. How can I further narrow down the strategies into an optimal one? If I take the best strategy (most profit over the 9 months of out of sample data) I worry that it is likely super over fit, and if I take the average of each parameter for each the 20k strategies I worry that this set of parameters may not work well together. How can I narrow down the ~20k strategies into one that works well and is likely not to be over fit? Thanks for your help. ## Answer by Unknown Coder (score 2) https://quant.stackexchange.com/a/14843 I agree with the previous statement that this is more stats related than anything else (it's not quant finance). But it's still a great question! This sounds awfully similar to linear regression testing with multiple predictor variables; you're basically doing it in a "monte carlo" fashion :) Depending on how your data is formatted, you could enter it into a program like Minitab and get the regression model from it within seconds. The model (or the resulting equation) would already give you the "best fit" that you are looking for. It would also give you a quantifiable number (like the resulting R-squared value) to give you a measurement of how accurate your strategy would be at any given time. ## Answer by Shahar (score 1) https://quant.stackexchange.com/a/14765 Choose the most robust (or insensitive) strategy. You are right that the best strategy might be overfit. So look at your parameter space and focus on the area where profitability, for example, changes least when you change the parameter value. Here is a 1D example: The most profitable strategy is that single point that unfortunately leaves no room for error - miss it and you fall from the abyss. However at the center of the graph, you will find the strategies that might be less profitable, but are much more robust: even if you missed a little bit, you would still be riding high. To be sure, this gets complicated in five dimensions - but if you look carefully into the data that you've created, you might find these robust areas. Good luck!
Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.