Choosing In-Sample and Out-of-Sample Windows for Strategy Validation

# What is the ideal ratio of in-sample length to out-of-sample length?


# What is the ideal ratio of in-sample length to out-of-sample length?












Suppose you are running a portfolio of quantitative strategies and that you develop a new potential strategy to be added to the mix. Assume for simplicity that the new strategy is independent of the existing strategy. The new strategy relies on data which is available going back X years. You proceed by backtesting and optimizing the parameters of the new strategy on an "in-sample" portion of your dataset, while reserving an "out-of-sample" portion for validation. The new strategy's weight in your portfolio will be determined by its out-of-sample performance. Your goal is to maximize your overall Sharpe ratio. What is the ideal ratio of in-sample length to out-of-sample length?

## Answer by vonjd (score 10, accepted)

https://quant.stackexchange.com/a/1481

Interestingly enough there is no scientific theory that suggests what fraction of the data should be assigned to training and testing and results can be very sensitive to these choices.

From Quantitative Trading by Ernest Chan (p. 53-54):

> Out-of-Sample Testing Divide your historical data into two parts. Save the second (more recent) part of the data for out-of-sample testing. When you build the model, optimize the parameters as well as other qualitative decisions on the first portion (called the training set), but test the resulting model on the second portion (called the test set). (The two portions should be roughly equal in size, but if there is insufficient training data, we should at least have one-third as much test data as training data. [...]

For more sophisticated methods see Evidence-based technical analysis by David Aronson p. 321-323.

I would add that once the strategy has been revised to reflect such data it is no longer "out-of-sample" or in other words: If you optimize your strategy "out-of-sample" you will incur data snooping bias via curve fitting nevertheless! Or as Aronson puts it:

> the Virginal status of the data reserved for out-of-sample testing has a short life span. It is lost as soon as it is used one time.

## Answer by B Seven (score 3)

https://quant.stackexchange.com/a/1987

I think it depends on many factors: the characteristics of the market, the consistency of the market, the number of parameters, the optimization criteria and methodology, the time frame (whether it trades weekly, hourly, by minute, etc), and even the trading strategy.

One approach to answer this question would be to test on many different assets and compare the results of different in sample/out of sample ratios. For example, if you have a system that trades large cap stocks, then maybe you could test different in sample/out of sample ratios on 100 large cap stocks. Then analyze the average and distribution of Sharpe Ratios for each in sample/out of sample ratio.

I would be looking for not only high average Sharpe Ratio but few low outliers.

By the way, due to the many problems of in sample/out of sample testing, I wonder if there are other approaches that have higher likelihood of producing profitable systems.

## Answer by babelproofreader (score 0)

https://quant.stackexchange.com/a/1999

Another approach is to use k fold or leave one out cross-validation by splitting the time series into a series of "chunks" according to the k value you want to use.

## Answer by Ishan Shah (score 0)

https://quant.stackexchange.com/a/44280

This is a good question. Rather than randomly selecting the ratio, it should be selected based on below two criteria.

- Maximum out of sample testing to check the effectiveness of the strategy

- Maximum in sample testing so that the machine learning model has sufficient data to learn from and at the same time there shouldn't be overfitting.

The first goal can be achieved by using K fold cross-validation. In these, the full data set is available for out of sample testing. This method splits your dataset into K equal or close-to-equal parts. Each of these parts is called a “fold”. For example, you can divide your dataset into 4 equal parts namely P1, P2, P3, P4.

The first model M1 is trained on P2, P3, and P4 and tested on P1. The second model is trained on P1, P3, and P4 and tested on P2 and so on. In other words, the model i is trained on the union of all subsets except the ith.

The performance of the model i is tested on the ith part.

When this process is completed, you will end up with four accuracy values, one for each model.

To achieve the second goal you need to keep sufficient data for backtesting. This will vary from applications to applications. But generally, you can keep 3/4th or 75% of data for training.

Thank you!