Evaluating Portfolio Variance Forecasts Beyond In-Sample Fit
Summary
The document asks whether a portfolio optimizer is useful when its in-sample variance is a better guide to test-period variance, even though its out-of-sample variance is higher than that of a Global Minimum Variance portfolio. The example contrasts the two methods’ training and test variances and highlights the gap between them as a possible measure of forecast quality.
The response cautions that a smaller train-to-test gap alone does not establish better performance. A high training error may reflect underfitting, and comparisons can be misleading if the estimators or loss measures are not on comparable terms. It recommends testing across multiple empirical and simulated datasets with varied distributions, means, variances, sample sizes, and asset counts. The discussion offers evaluation guidance rather than evidence that the proposed optimizer improves returns; conclusions may depend on the datasets and conditions examined.
Key ideas
- A smaller gap between training and test variance does not by itself show that an optimizer performs better.
- Compare methods using a consistent loss measure on out-of-sample weighted returns.
- A high training error paired with a lower test error may indicate underfitting or a favorable sample.
- Repeat evaluations across datasets with varied distributions and portfolio dimensions.
Tags
Full text
# Benefit of better predicting the variance of portfolio daily returns while optimizing a portfolio? # Benefit of better predicting the variance of portfolio daily returns while optimizing a portfolio? ### Question - Is there a benefit of having lower gap between 'in-sample' variance of portfolio daily returns and 'out-of-sample' variance of portfolio daily returns? (= better estimates the out-of-sample variance) ### Question in more detail - I have developed a way of optimizing a portfolio, based on Global Minimum Variance portfolio optimization. - There are upside and downside of my portfolio optimization method. cons : It cannot lower the `out-of-sample variance' of portfolio daily returns than the GMV portfolio. In other words, my portfolio optimization method fails to achieve better portfolio performance such as Sharpe ratio. pros : However, my portfolio optimization method has lower gap between 'in-sample variance' and 'out-of-sample' variance than the GMV portfolio. - For illustration, let me give you an example. During training period to come up with how much weight to put on each stock, GMV portfolio optimization calculates the stock weight with variance 100 of daily returns. However, during the investment period (test period), it gives me 125 for 'out-of-sample' variance. - My portfolio method gives me 150 variance for 'in-sample variance' and 130 for 'out-of-sample' variance. As you can see, the actual variance is still low with the result of GMV portfolio optimization method. However, my method expects the 'out-of-sample' variance better than GMV method. GMV method is wrong by 25 percent, while my method is wrong by 13 percent. - As such, I am curious to know if my portfolio optimization method would be useful in any case of trading in stock market nowadays. ## Answer by develarist (score 1) https://quant.stackexchange.com/a/49707 If the test (out-of-sample) variance for the new model is still higher than that of the classical model, then it does not offer outperformance, at least for the dataset used. Also, having a high training error and low test error from the new model by itself might mean it is underfitting the training data and somehow generalizing well to the test data, which must be explored more by trying more than one dataset. If there is a smaller difference between train and test variance in the new model than the difference found in the traditional model, first consider whether your new model's estimator of variance should even be compared to the traditional estimator unit-wise, otherwise find a different evaluation metric for loss/error computed from the weighted out-of-sample data instead that puts the new and old model on equal footing. In the case that you're not comparing between the new model and classical model, but instead only the new model's train and test results: If the test loss value is lower than the train loss value, this is a good thing, but could be an artefact, i.e. the data you're using is making the outcome ideal. Repeat the experiment on a variety of datasets, artificial and empirical, that have different distributions, means, variances, number of observations and number of assets. If the result is fairly consistent through all of these combinations, or if there are certain distributions that the results hold for and not others, then you're onto something.
Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.