Evaluating a Trading Signal and Checking Stock Data Quality
Summary
The document considers how to assess whether a time series signal predicts stock prices when no prediction formula or signal timing is supplied. It recommends first clarifying when the signal becomes available, then testing whether it predicts future outcomes rather than prices from the same date. Possible targets include multi-period close returns, price direction, or the high-low range. Suggested analysis includes visualizing relationships, comparing a fitted regression with a zero-return baseline, checking residuals and outliers, considering prior signal values, and evaluating on held-out data or with cross-validation.
For data quality, the response proposes checking missing observations, date order and gaps, plausible OHLC relationships, distributions of values and returns, extreme moves such as those caused by stock splits, and the signal's distribution. The sample is only a few weeks and lacks context about the signal, adjustment conventions, and forecast horizon, so it cannot establish reliable predictive performance by itself. These are exploratory suggestions rather than a defined protocol; any model selection and evaluation would need to avoid look-ahead bias and account for the limited sample.
Key ideas
- Clarify when each signal value is known before deciding which future market outcome it can predict.
- Test forecast horizons and targets such as future returns, direction, or intraday range.
- Compare fitted predictions with a simple baseline and examine errors on data not used for fitting.
- Check missing values, date continuity, OHLC consistency, unusual moves, and signal distributions.
- A short sample and unspecified signal construction limit conclusions about predictive power.
Tags
Full text
# evaluate the predictive power of a signal to predict stock price - interview question # evaluate the predictive power of a signal to predict stock price - interview question I am a young Statistics graduate. A few days ago as an interview question, I have been asked to evaluate the predictive power of a Signal time series (supposedly output by an Artificial Intelligence system) in predicting a stock price time series and I was provided with a sample data set as follows: | Date | Open | High | Low | Close | Adj Close | Signal | | 202106-01 | 627.80 | 633.80 | 620.55 | 620.55 | 623.90 | 85.11 | | 2021-06-02 | 620.13 | 623.36 | 599.14 | 620.13 | 605.12 | 76.59 | | 2021-06-03 | 601.80 | 604.55 | 571.22 | 571.22 | 572.84 | 68.73 | | 2021-06-04 | 579.71 | 600.61 | 577.20 | 600.61 | 599.05 | 78.47 | | 2021-06-07 | 591.83 | 610.00 | 582.88 | 610.00 | 605.13 | 78.63 | and so on for some few weeks. No further information have been provided. I then pointed out that without knowing the analytical formula of the function employed to predict the stock price from the Signal value Pred(Signal) = Stock Price Prediction it was not possible (at least for what I know) to calculate any prediction error and thus to evaluate the prediction power of Signal. The reply from the interviewer was that the problem is solvable as it is. FIRST Question: how could one evaluate the predictive power of Signal given only the data above? As a further question from the interviewer I was asked to "review the quality of the data, list any potential errors, and propose corrected values. Please list each quality check error and correction applied." Really I had no idea how to proceed in answering this Second Question. SECOND Question Anyone has any pointers on any possible data quality procedure that I could have applied? Thank you for your kind suggestions. ## Answer by Pontus Hultkrantz (score 2) https://quant.stackexchange.com/a/71089 You might find more traction at https://stats.stackexchange.com/, since most ML and data kind of problems are handled there. I am no ML expert but let me brainstorm for you, since this kind of task is slightly more an art than a science. You need be creative than just following a det of rules. > FIRST Question: how could one evaluate the predictive power of Signal given only the data above? This looks like a regression type task, predict k-steps ahead given current or past signals. Question that could have been asked if possible is whether the signal for a particular day is given before or together with the OHLC data. If so, then we want to se if the signal can predict future OHLC data (and not same row). The rest is the usual ML data procedures Example: - Explore data for validity etc. - Divide the data into train/test. - Normalize columns, and e.g. convert to k-step close price returns. - Explore e.g. cross correlation between signal and k-step close return. Maybe signal predict not close price return but high-low range? Maybe normalize close return by high-low range? Play around, visualize scatter plots. Easier to predict direction of the future price via e.g. logistic regression? - Make an e.g. linear regression for Close price return k steps ahead using the current signal. Try for various steps of k. How well does Signal predict Close Price k steps ahead of time: what does the error look like, are there outliers, do we need to change the fitting approach, robust? Does the model predict better than simply guessing zero return? If there is trend in the data, maybe subtract it first. - Do we need to use previous signals into consideration as well? - Does it make more sense to predict Close or Adj Close? Possibly the signal isn't taking adjustment (corporate actions) into account. - Evaluate on test set / perform cross-validation etc. > SECOND Question Anyone has any pointers on any possible data quality procedure that I could have applied? Some basic things I can think of - missing values? - assert 0<low<open<high and 0<low<close<high, dates are increasing without gaps, etc.. - Histogram of columns, histogram of changes in values (returns) etc. Any extreme values (e.g. stock split)? What is the distribution of the signal? - Normalization applied based on observation.
Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.