Evaluate Stock Predictors Separately from Trading Strategies
Summary
The note considers how to compare stock ranking models intended to surface candidates for later fundamental analysis. Its central recommendation is to define the predictor and its outcome separately, then test whether the predictor ranks or classifies future returns as intended. For example, a model based on a stock’s position within an estimated price range could be assessed by comparing range-position groups with subsequent return groups, using rank correlation as one possible measure.
The proposed evaluation avoids adding entry rules, stop losses, or exit management when the goal is to compare forecasting models. Combining those trading rules with the predictor makes it difficult to tell which component produced the results and can encourage fitting to historical data. The note does not settle which benchmark, forecast horizon, or performance measure is best for every use case; those choices should follow the specific prediction being tested. It gives a general evaluation principle rather than empirical results or a full backtesting protocol.
Key ideas
- Define the prediction target and forecast horizon before scoring a stock model.
- Evaluate the predictor against its stated outcome rather than a complete trading strategy.
- Rank correlation between predictor groups and later return groups is one possible measure.
- Adding stops and trading rules can obscure predictor performance and increase overfitting risk.
Tags
Full text
# strategy for backtesting # strategy for backtesting Short version: How do I evaluate models that predict stock performance? Back story: I'm working for a stock analyst, writing an application to sort through stocks and find candidates that fit the criteria he is looking for. The point is to bring stocks to his attention so he can do fundamental analysis. For example, he has a mechanism for calculating the expected price range, then we look at where it is currently in that range. There are various factors that might put pressure on the stock to move up or down. Each of his models gives the various factors a different weighting. We want to backtest the models against the last ten years of data, including the recession. For the top and bottom 25 stocks for a given model projection, I need to score how the stocks did over the next period. Repeat. I cumulatively score the models and see which give the best predictions. My question is how best to score the stock performance. Do I take into account that they should sell if the stock price goes down e.g. > 10% of the price range or that the stop-loss should ratchet up? But those have more to do with the buying/selling scheme, nothing to do with the projection model. In this case, I just want to evaluate model against model. Do I compare price rise to change of the entire market? Or just within industry? Or compare against the price range for that stock? There are lots of sites that allow you to do backtesting, but that doesn't answer my question about how to score the models. The article on evaluating Bollinger Bands comes the closest of anything I've seen to touching on the topic but still manages to avoid it. ## Answer by experquisite (score 1, accepted) https://quant.stackexchange.com/a/33110 Split the strategy into predictors and outcomes, then evaluate how well the predictor predicts the outcome. Don't backtest a trading strategy, evaluate an estimator/predictor using typical machine learning/stats. If the position within a range is supposed to predict whether a stock will go up or down after 10 days then just test that; measure the rank correlations between range decile and 10 day future return decile or whatever. Don't try and mix an entire trading strategy and backtest repeatedly; you won't know what you are measuring and you'll overfit to the historical data.
Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.