Choosing Forecast Models When the Test Sample Is Very Small
Summary
The document considers how to choose between two forecasting models for a multifamily rental vacancy rate when the available history is used for training and only the final two observations are held out for testing. The first model has better reported training accuracy, while the second has better test errors on several measures. The question is whether to prioritize the test set despite its small size or favor the model that fits the training data more closely.
The answer describes model selection as a balance between bias and variance, noting that aggregate error measures alone cannot reveal which problem dominates. It recommends inspecting forecast plots and choosing with attention to the consequences of being wrong. It also treats disagreement between forecasts as useful information about uncertainty and stresses the practical cost of overestimating versus underestimating vacancy. The response offers judgment rather than a statistical selection procedure; with so few held-out observations, the reported test comparison is highly limited and does not establish which model will forecast best in future periods.
Key ideas
- Training accuracy and test accuracy can favor different forecasting models.
- A test set containing only a few observations provides weak evidence for selecting a model.
- Aggregate error metrics do not by themselves distinguish model bias from estimation variance.
- Forecast plots and the business cost of different errors can inform a practical choice.
- Disagreement between forecasts can help communicate uncertainty about future outcomes.
Tags
Full text
# How to interpret the accuracy result of the forecaste?
# How to interpret the accuracy result of the forecaste?
I'm trying to forecast the vacancy rate of multifamily rental property. I have the data from 1992 until today. I'm trying to fit a model with the serie without the last 2 observations.
I only need forecasting for 2 years into the future.
My question is : Which model is better in forecasting?
The first model seem to do a better job with the training set than the second model. But the reality seem to be reversed when we look at the test set of only 2 periods.
I think I should look at the test set, but at the same time, I wonder if it could be only better by chance since the test set is comprised of only 2 observations.
Should I look at the training set or at the test set? Which model will you choose and why?
Thank you in advance!
I'm using R.
> accuracy(fit1, ti_canada[26:27])
```
ME RMSE MAE MPE MAPE MASE ACF1
```
Training set 0.00944111 1.1486731 0.8727625 -13.108775 28.172474 0.7247854 -0.0850986
Test set -0.19295099 0.5297774 0.4933903 -3.432013 8.200891 0.4097359 NA
> accuracy(b6, ti_canada[26:27])
```
ME RMSE MAE MPE MAPE MASE ACF1
```
Training set 0.09192687 1.3890104 1.1561171 -9.349635 39.581546 0.9600973 0.387796
Test set -0.31354649 0.3881324 0.3135465 -5.268731 5.268731 0.2603846 NA
## Answer by AlainD (score 0)
https://quant.stackexchange.com/a/40845
The question of choosing the best model, is a matter of balance.
All your indicators (ME, RMSE, MAE, MPE, MAPE, MASE, ACF1,...) are aggregations of two types of errors : a bias (you have the wrong model but an accurate fit) + a variance (you have the right model but a inaccurate fit). And there is no statistical method to know if you have a high bias and low variance or a high variance and low bias.
So I suggest, you make a plot and make an eye-stimate to select the "best" one, best meaning with the least business consequences if you are wrong.
Furthermore, you should rejoice. Having two forecast gives you an idea of how you can be wrong. Because, face it, you will be wrong! What maters is not to be very far off.
Knowing in advance the precision may be reassuring. But on the bottom line, where would you loose more money: by underestimating or overestimating the vacancy rate ?Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.