Interpreting In-Sample and Out-of-Sample R-Squared

# R-squared to be computed on training sample or test sample?


# R-squared to be computed on training sample or test sample?












I am currently going through the book Machine Learning For Factor Investing whose online version can be read here: http://www.mlfactor.com

In the section on model validation, one can read the following

> In line with the linear regressions, the $R^2$ can be computed in any predictive exercise. $$R^2 = 1 - \frac{\sum_{i=1}^I(y_i-\tilde{y_i})^2}{\sum_{i=1}^I(y_i-\bar{y})^2}$$ where $\bar{y}$ is is the sample average of the label. One important difference with the classical $R^2$ is that the above quantity can be computed on the testing sample and not on the training sample. In this case, the $R^2$ can be negative when the mean squared error in the numerator is larger than the (biased) variance of the testing sample.

What does the author mean by classical $R^2$? More importantly, why does he say the quantity cannot be computed on the training sample?

## Answer by Dave (score 1)

https://quant.stackexchange.com/a/69269

Only under specific circumstances does $R^2$ have its traditional interpretation of the proportion of variance explained, and out-of-sample is not such a circumstance. However, some kind of standardized measure for regression performance could be more helpful than saying, “Well, boss, we’ve got an MSE of 11.”

(I’m not actually convinced that this is as helpful as it sounds. For example, is $R^2=0.4$ always like an F-grade in school, or is $R^2=0.9$ always like an A-grade?)

To answer your question, consider what $R^2$ does in its traditional setting.

$$ R^2 = 1-\dfrac{\sum\bigg( y_i-\hat y_i \bigg)^2} {\sum \bigg( y_i-\bar y \bigg)^2 } $$

Looking at the equation, $R^2$ compares the performance of your model that makes predictions $\hat y_i$ to a naïve model that, in order to estimate the conditional expected value, always predicts the pooled/marginal expected value of $y$, so the mean $\bar y$.

Therefore, to do some kind of out-of-sample $R^2$, the only way that, to me, fits with the spirit of that is to compare your out-of-sample predictions to the predictions of a naïve model that always guesses the same value: the mean of $y$ as calculated from the in-sample data, so $\bar y_{in}$ and not $\bar y_{out}$.

Explicitly answering your two questions:

- The “classical” value is the usual in-sample $R^2$.

- I think there is an implied “just”, as in “not JUST on the training sample”.

EDIT

I will include an `R` simulation to show that $R^2$ loses its interpretation as "proportion of variance explained" when out-of-sample data are used, since the "other" term in that Cross Validated link does not equal zero.

```
set.seed(2021)

# Set size of both training and test sets
# (They're not usually equal, but they will be here,
# since simulation data are unlimited)
#
N <- 1000

# Simulate the in-sample and out-of-sample values of a feature
#
x_in <- runif(N, 0, 7)
x_out <- runif(N, 0, 7)

# Simulate the in-sample and out-of-sample values of a response variable
#
y_in <- x_in + rnorm(N)
y_out <- x_out + rnorm(N)

# Fit a linear regression to the in-sample data
#
L <- lm(y_in ~ x_in)

# Make predictions on the in-sample and out-of-sample data
#
preds_in <- predict(L)
preds_out <- predict(L, data.frame(x_in = x_out))

# Calculate that "other" term from:
# https://stats.stackexchange.com/questions/551915
#
other_in <- sum((y_in - preds_in) * (preds_in - mean(y_in)))
other_out <- sum((y_out - preds_out) * (preds_out - mean(y_in)))

# Also calculate the "other" term using the out-of-sample mean of y
#
other_out_out_mean <- sum((y_out - preds_out) * (preds_out - mean(y_out)))
  
# Display the "other" terms to show that the in-sample "other
# is effectively zero while the out-of-sample "other"
# has a large absolute value of ~50.
#
# Using the mean of out-of-sample y does not fare much better,
# since the regression coefficients are OLS estimates for the
# in-sample data, NOT for the out-of-sample data
#
other_in # 1.226068e-11
other_out # -50.96088
other_out_out_mean # -51.3533
```
```