Using ROC and Classification Metrics to Evaluate Trading Rules
Summary
The document considers whether a technical trading rule for cryptocurrency can be assessed with receiver operating characteristic analysis. The proposed labeling marks a buy signal as correct when the underlying asset has a positive return that day and incorrect when the return is negative. The response explains that this setup produces discrete predictions, which are not sufficient to trace a conventional ROC curve; ROC analysis requires a continuous score or probability that can be thresholded.
For binary buy outcomes, it recommends examining a confusion matrix and measures such as precision, recall, and F1 score by comparing forecast labels with realized outcomes. It mentions an example involving a trend-following rule combined with a secondary model, while clarifying that the example’s predictions were all labeled positive and therefore illustrate a special setup. The discussion does not provide empirical performance evidence or address trading costs, return magnitude, or risk-adjusted profitability, so classification quality should not be mistaken for strategy value.
Key ideas
- A conventional ROC curve requires a continuous score that can be evaluated across thresholds.
- Discrete buy or no-buy labels can instead be summarized with a confusion matrix.
- Precision, recall, and F1 score compare predicted outcomes with realized labels.
- The proposed same-day positive-return label does not measure return size or trading profitability.
- The cited strategy example includes a secondary model and has a specialized labeling setup.
Tags
Full text
# Assessing goodness of a Technical Trading Rule using a ROC model # Assessing goodness of a Technical Trading Rule using a ROC model I am testing various technical trading rules (TTR) on the cryptocurrency market. I have already setup some significance tests, to compare the returns and volatilities. I would now like to test it from a different angle using the ROC model, often applied when backtesting PD [Probability of Default] models. I am just not sure if this is actually feasible in the environment I am trying to apply it. The logic I have applied until now is that if the output of my TTR is a "BUY" and the underlyings returns that day is bigger than zero, it is a TRUE BUY on the other hand if the return of the underlying is smaller than zero that day I get a FALSE BUY. Having this information I think I should be able to set-up a ROC model. My question here would be, is this approach actually feasible? Has someone already taken this approach? ## Answer by Jacques Joubert (score 1) https://quant.stackexchange.com/a/44939 So ROC performance metrics are common place in machine learning applications, particularly for classification tasks. Looking at the way you have setup the question here you wont be able to plot a ROC curve as it relies on a continues output (not discrete). You would however be able to plot a confusion matrix and look at performance metrics such as precision, recall, and F1-score. All you would need is the actual forecast vs your predicted forecasts. I did this in python using Sklearn on a simple trend following strategy: Note: This strategy I built here uses a secondary model to determine when to place a trade based on the primary technical model and that is why all of predictions are labeled as 1. It does however still illustrate the point.
Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.