How Backtesting, Simulation, Bootstrap, and Cross-Validation Differ

# Difference between cross-validation, backtesting, historical simulation, Monte Carlo simulation, bootstrap replication?


# Difference between cross-validation, backtesting, historical simulation, Monte Carlo simulation, bootstrap replication?












To determine if a strategy is better than others, or to optimize the parameters of a model, the following statistical techniques are often employed, often one over the others instead of altogether. Their results are important in terms of training a model or strategy and ensuring it will retain predicted performance when applied to unseen test data, but what are all the differences between their procedures as well as weaknesses, applicability and strengths, given that many seem to do their own rendition of data resampling? To start, a brief description of their procedure might help for comparison.

- Backtesting

- Historical simulation

- Monte Carlo simulation

- Bootstrap replication

- Cross-validation

## Answer by Attack68 (score 4)

https://quant.stackexchange.com/a/50040

As with many things, particularly in machine learning and AI, I think you will find that these processes do not have a unique, logically or mathematically defined description. More so I would say that depending upon the context they can mean different things and might even mean the same thing. However, in my experience this is their most common usage.

#### Backtesting

In machine learning I have come across 'backtesting' in the domain of finance when a model or strategy has been defined and its purpose is to genuinely test the profitability of the actions on some real historical data. Therefore backtesting requires time series and real data and aims to test a model. If the model has been designed with the test data in mind, or has snooped then the test will not be reflective of real performance.

#### Historical Simulation

This is similar to the above except I would say it falls in a more general context. Not just necessarily when you have a profitability strategy to examine but possibly just to examine risk, or other factors, as well. For example a historical simulation of Value at Risk of a portfolio. Historical simulation requires time series and real data.

#### Monte Carlo

Monte carlo is the same as the above but rather than requiring real data it uses simulated data. How the simulator is defined will determine the success of the analysis, e.g. parametrically, non-parametrically or another random process. Monte carlo is used for a multitude of tasks not necessarily in finance or time series related.

#### Bootstrap Replication

Bootstrap replication is, I would describe, between the two above models. Bootstrap samples create a statistical sampling methodology where the underlying real data is used to some degree. Either it is repetitively sampled (non-parameterically) or a parametric model might be created from it which generates samples from a probability distribution. Although this might be quite similar to Monte Carlo, I think by definition this will be more closely related to the underlying historical data.

#### Cross Validation

When training a machine learning model there are often two types of parameters to determine: basic parameters and hyper parameters. Basic parameters are the underlying values needed for the model, for example a linear regression model needs coefficients. These are trained from the training data. However, when you train a model on the data itself it is very biased, and might be able to reproduce that data exactly. But when tested against new and unseen data it might perform very poorly. Therefore you often need hyper parameters to be trained to analyse how effective a 'trained' model can be on unseen data. Hyper parameters might be things how many nodes to use in a neural network, or how many clusters to use in k-means clustering. You might even consider it a hyper parameter to decide whether to use SVMs or Logistic Regression or a Decision Tree, for example. Cross validation often uses clever mixes of the data you have to determine a good set of basic parameters and hyper parameters. You can then test the final model on test data.