Why Stock Prices Forecast Well While Returns Do Not
Summary
The discussion explains why models can track a stock’s closing price closely while failing to predict its returns. Because consecutive prices tend to be similar, a regression using recent prices can produce predictions near the next close. That visual fit does not mean the model has found a useful trading signal: its error largely reflects the daily price change, which is the quantity a trader needs to anticipate. The discussion frames this as a consequence of prices behaving approximately like a random walk, where the next price is expected to be near the current one plus an unpredictable change.
The example concerns Netflix prices and compares linear regression and gradient boosting on closing prices and returns. It gives no formal backtest or evidence of profitable trading. The main lesson is to evaluate forecasts against the target that matters for the decision, rather than relying on close-price fit alone. The random-walk account is an approximation; the response allows that price predictability may appear briefly in some situations.
Key ideas
- Consecutive closing prices can be highly correlated even when returns are difficult to predict.
- A close-price model may appear accurate simply because its forecast stays near the previous close.
- The forecast error may largely represent the price change that matters for trading.
- Good fit on price levels does not establish that a model can produce a useful trading signal.
- Approximate random-walk behavior does not rule out temporary predictability in some market conditions.
Tags
Full text
# Closing prices are predicted very well but returns are predicted poorly
# Closing prices are predicted very well but returns are predicted poorly
I'm learning some time series analysis and forecasting techniques, I've tried to predict stock prices for Netflix but I'm very confused.
At first I've tried Auto ARIMA which gave me a straight line, obviously it's a bad fit, then I tried a linear regression between X(t) and it's lagged version, I've plotted a lag plot and saw that there is a very strong correlation between X(t) up to X(t-10) so I trained a linear regression model using X(t-1)...X(t-6) as features (predictors) and X(t) as a target.
I've compared the predictions next to the test set and the results were quite shocking, the model was nearly perfect and predictions were almost equal to actual values in the data set.
The MAE is only 6.25 (6.25 dollars off in average).
Next I tried another ML technique which is the Gradient Boosting Trees algorithm and results were as perfect as the linear regression model, you can see the results here
So I was thinking that something was wrong and I tried changing my variable, this time instead of using closing prices I used returns (using both algorithms) and the results were very bad and very off as you can see here:
and this is when I multiply predictions by 10:
These results are very confusing for me, I'm wondering why am I fitting the closing prices almost perfectly while returns are modeled quite badly ? and most importantly What's the recommended approach to predict stock prices ?
Note: I already know that returns are stationary while closing prices tend to not be, but is this important ? and If so why ?
Thank you !
## Answer by Lliane (score 5)
https://quant.stackexchange.com/a/53438
The previous day price is an excellent predictor of the next day price, however the previous day return doesn't tell you much about the next day return.
## Answer by Ivan (score 1)
https://quant.stackexchange.com/a/53451
A stock price in theory is a random walk: the expected value of today’s price is yesterday’s price. So from looking at a chart, yes a regression of $S_t$ on $S_{t-1}$ will appear very close in terms of fit. That’s because the price today is that of yesterday plus or minus a random element centered around zero.
That’s not to say that regression is useful as a trading tool. Your regression error will basically be your daily change in price which is the actual quantity of interest in terms of trading. So you haven’t extracted information that is going to be of use for the task at hand.
Of course in reality some prices some of the time will not be true random walks and those are times when a simple day-on-day price regression may help in your trading. This should be exceedingly rare and fleeting though.Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.