Evaluating a 58% Stock Direction Forecast Beyond Accuracy
Summary
The document considers whether a technical signal that predicts stock direction correctly 58% of the time is useful. Its answers caution that raw hit rate is difficult to interpret because stocks have an upward drift, and a forecast must be compared with a suitable baseline over the same period. Directional accuracy also ignores the sizes of gains and losses, so a strategy can lose money if its losing trades outweigh its winners or if trading costs and implementation erode returns.
The suggested evaluation process is to fit or tune a model on one part of the data, then assess it on separate out-of-sample periods, potentially using rolling windows. One response recounts a machine-learning example with strong in-sample accuracy that fell to chance out of sample, illustrating overfitting risk. The document offers no universal accuracy threshold; profitability depends on the benchmark, return distribution, costs, and how the signal is traded.
Key ideas
- A 58% directional hit rate needs comparison with the stock’s baseline tendency and an appropriate benchmark.
- Out-of-sample evaluation helps reveal whether a signal generalizes beyond the data used to build it.
- Hit rate alone omits the size of wins and losses, which determines the strategy’s return profile.
- Transaction costs and implementation can make even a directionally accurate forecast unprofitable.
- Strong in-sample performance can deteriorate substantially on later data.
Tags
Full text
# If I have found a way to predict stocks trend with 58% accuracy, is it good? # If I have found a way to predict stocks trend with 58% accuracy, is it good? Say I have found a way through technical analysis to predict how stocks would behave with 58% accuracy, how good is this percentage? ## Answer by vonjd (score 4, accepted) https://quant.stackexchange.com/a/18533 In the long term you will underperform buy & hold because you need an accuracy of at least 65%. See these papers for more: - Bauer, R.; Dahlquist, J.: „Market Timing and Roulette Wheels Revisited“, CFA Institute, 2012. http://www.cfapubs.org/doi/pdf/10.2469/irpn.v2012.n1.10 - Sharpe, W.: “Likely Gains from Market Timing”, Financial Analysts Journal, March/April 1975. ## Answer by pat (score 4) https://quant.stackexchange.com/a/18531 It's not unusual to find a financial time series with positive trend samples biased between 55-60%, depending on the period sampled. Stocks tend to have an upward drift over the long run. When you account for the drift, I would say, that number is really not much better than chance. A better way to verify your question would be to make certain to build your estimates and models (with parameter optimization) on a certain portion of the sample and comparing out of sample results of your model's performance to the actual performance of the stock series under the same out of sample time frames. Using a rolling window is commonly used. And also, make sure to look at the actual expectation or performance of the actual returns over the comparison period, as simple directional signs can be close to meaningless without values for context. Incidentally,this researcher tested 1 million machine learning models using data from 93 - 2008 to predict SPY ETF direction and acheived an astounding 78% in sample performance. When the trained model was tested over the next two years out of sample data, it dropped all the way to 50%... not so remarkable, and a good example of why its really important to consider validating data over different periods. ## Answer by sanliusinger (score 4) https://quant.stackexchange.com/a/18536 It's not bad but you have to backtest the method out-of-sample. Say you have discovered an indicator that works 100% in history, you still cannot be sure if it works next time. Another advise is you might want to investigate the distribution of loss when your system fails to work. If your system delivers 1% every time you trade, and loses 10% each time it fail, you probably wind up losing in the end. ## Answer by madilyn (score 2) https://quant.stackexchange.com/a/18528 Directional forecast is insufficient. You could have a signal that has 100% accuracy and you would not necessarily be able to profit from it because of transaction cost, implementation etc.
Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.