Bootstrap Methods for Testing and Tuning Trading Models
Summary
The discussion asks whether tuning a trading model across bootstrapped versions of historical returns can reduce overfitting to one observed market path. It outlines dependent bootstrap approaches that preserve time-series structure, including resampling contiguous or random-length blocks, resampling residuals from a fitted model, and simulating from a model’s residual distribution. Conditional generative models are also mentioned as a way to create challenging synthetic series.
Resampling is used in evaluating whether strategy performance is statistically significant when many candidate strategies have been tried. The answer connects this to multiple-hypothesis testing and model selection, while noting that combining models may perform better than tuning on synthetic data alone. A second response sees simulation as more directly useful for risk estimation and warns that synthetic returns inherit assumptions from the model that generated them. Thus, resampling may support robustness checks, but it cannot make artificial market paths equivalent to new observed evidence.
Key ideas
- Dependent bootstrap methods aim to preserve important time-series properties such as autocorrelation.
- Block, stationary, model-based, and Monte Carlo approaches generate alternative return paths in different ways.
- Resampling can help assess strategy performance amid multiple testing and model selection.
- Synthetic data for alpha tuning inherits the assumptions and limits of its generating model.
- Ensemble methods may be more effective than relying solely on synthetic-data fine-tuning.
Tags
Full text
# Do you optimise models on bootstrapped time series? # Do you optimise models on bootstrapped time series? As Quants, we soon learn to optimise models, by fitting them to historical time series, e.g. the historical daily returns of some stock. But the historical series of daily returns is just one realisation, out of many possibile series that could originate from the same distribution of daily returns. So if I fit my model to a specific realisation of daily returns -which happens to be the historical series- I might be overfitting the model. Wouldn't it be more correct to optimise the model, based on N boostrapped series of daily returns, all originating from the actual historical series? Where would this technique lie, on the spectrum from stupid to industry standard? 😋 ## Answer by NBF (score 5, accepted) https://quant.stackexchange.com/a/44104 Simulation for timeseries data is not a trivial matter and there are a number of methods to ensure you retain some of the relevant properties (mostly called dependent bootstrap methods): - Block bootstrap - contiguous blocks of data chosen so that they are large enough to retain significant autocorrelations. - Stationary bootstrap - randomised block size - Model based bootstrap - fit model (eg ARIMA) and bootstrap residuals (IID is theoretically ok). - Monte Carlo - fit model and simulate from theoretical distribution for residuals. - conditional GAN - condition on relevant time-series data and generate data which challenges an appropriate discriminator. Many of these methods are used for generating the necessary statistics to test whether one of many possibles strategies has a statistically significant return/Sharpe/other performance measure. White’s Reality Check test and its many variants (by Romano-Wolf and by Hansen et al-Model Confidence sets) test between many models using this multiple-hypothesis testing framework. This is a type of model selection and bears a strong resemblance to hyper-parameter fine-tuning. In our paper GAN for Trading Strategies, we try to do a relatively comprehensive literature review and to use cGANs to generate more data for fine-tuning. We find that, in general, model combination / ensemble methods work better. Regardless,the notion of generating new data for more robust predictors / better out-of-sample performance is quite reasonable. ## Answer by DoctorOrrery (score 0) https://quant.stackexchange.com/a/43563 It is a useful technique, but more for risk rather than return estimation. I believe Markowitz was working generating entire market simulations down to individual market participants in order to get a better idea of market dislocations. If you use Ledoit-Wolf for covariance shrinkage it can involve bootstrap simulation of factor IC series from random sample covariance matrices. The problem with using it to optimise alpha models is that you're going to be using some semi-deterministic model of stock returns to generate your return series. If you then try to compute a model to predict those returns, the best you could do is to discover your semi-deterministic model, which is an approximation rather than the real market.
Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.