Separating Forecast Accuracy from Trading Strategy Performance
Summary
The discussion asks how to evaluate a stock price forecasting algorithm separately from the trading rules that convert forecasts into trades. One response suggests repeatedly backtesting over shifted date intervals and averaging measures such as net profit and winning-trade percentage. This evaluates the combined forecast-and-strategy system, however, so it does not isolate forecast quality by itself.
Another response distinguishes computational speed from investment performance. It proposes using averaged historical data to reduce the amount of history needed for near-real-time calculations, while noting that market outperformance depends on the forecast model and its parameters. The thread offers no controlled comparison, formal forecasting metric, or empirical results. Its main limitation is that it leaves the central decoupling problem unresolved: strategy-based performance measures remain sensitive to trading rules, and the proposed software dependency layer addresses parameter connections rather than a standalone measure of predictive value.
Key ideas
- Shifted-window backtests can show how combined strategy performance varies across date intervals.
- Net profit and winning-trade percentage are strategy outcomes, not pure measures of forecast accuracy.
- Averaging historical data may reduce computation when processing incoming market data.
- Investment performance depends on the forecasting model, its parameters, and the strategy that uses its outputs.
Tags
Full text
# Evaluating forecasting algorithm # Evaluating forecasting algorithm I am trying to evaluate a forecasting algorithm for stock price prediction. However, the performance of the algorithm may be very much tied to the trading strategy. Is there a systematic way for decoupling these two, and evaluating the forecasting algorithm only? Perhaps, something not related to MSE, as it does not give much of an idea on how well or bad the algorithm will do about return of investment. ## Answer by IgorS (score 1) https://quant.stackexchange.com/a/4360 Are you talking about forecast performace? You should do a number of back tests on the intervals shifted by 1-2 weeks (or days, depends on trading frequensy you use) and collect the average of all "back test vs market" results. NetProfit, WinTrade percent and any other statistics could be used as a performance metrics. ## Answer by thwd (score 0) https://quant.stackexchange.com/a/4144 If you're talking about the performance as the speed in which the software algorithm can be executed on your hardware, then my suggestion would be to work with an averaged historical-data array. This way you can still make realtime or close-to-realtime calculations on incoming data feeds without having to consider the whole past data-density, while staying on a close-to-zero deviation from the effective values. If you're talking about the performance as in 'outperforming the market' then your problem is tightly dependent on the forecasting algorithm itself and the parameters it takes to operate. The only viable way to decouple the two would be to create a bold dependency-management layer between your strategy and the algorithm that allowed its user to connect arguments from their strategy to parameters of the forecasting algorithm. This will very probably have a strong influence on the speed in which the algorithm can execute, because of all the indirection and checks that the system has to perform to remain stable and rolling.
Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.