Trading Classifiers: Thresholds, Expected Value, and Meta-Labeling

# Defining an objective function for machine learning task of trading


# Defining an objective function for machine learning task of trading












A simplified example. Given:

- asset's price time series

- fixed distances to stop and target.

A function of these inputs has two possible output values: $1$ if price is likely to hit the target earlier than stop and $0$ otherwise. This function is implemented by means of machine learning, e.g. a neural network.

For reasonably and equally sized target and stop, the feasible win rate will be close to $\frac{1}{2}$. Most of times the outcome is not predictable and the right choice is not to open a position. From time to time the odds are marginally better than $\frac{1}{2}$; this is when I want the algorithm to output a $1$. Cherry picking.

When putting this up as a binary classifier, it appears that the widely used binary cross-entropy loss

- strongly penalizes confident mis-classifications

- weakly rewards being rightfully more confident

and is not necessarily appropriate here.

To be trainable by means of stochastic gradient descent, this neural network needs to produce a continuous output $\in(0;1)$, and the objective function must have a useful derivative. This is why simply counting winners and losers wouldn't help. The objective function needs to weigh network's confident and less confident decisions differently.

How would you deal with it?

Here is a similar question, probably better written.

## Answer by Alexey Golyshev (score 7, accepted)

https://quant.stackexchange.com/a/44212

The code below is written in Wolfram Mathematica.

For example, we have some training data. And we are trying to predict: long (1) or short (0).

```
SeedRandom[0];
n = 10000;
X = RandomReal[{-1, 1}, {n, 100, 5}];
Y = RandomInteger[{0, 1}, n];
```

```

net = NetChain[
  {
   LongShortTermMemoryLayer[64],
   SequenceLastLayer[],
   ElementwiseLayer[Ramp],
   LinearLayer[2],
   SoftmaxLayer[]
   },
  "Input" -> {100, 5},
  "Output" -> NetDecoder[{"Class", {0, 1}}]
  ]
```

```
SeedRandom[0];
netT = NetTrain[
  net,
  X -> Y,
  All,
  LossFunction -> CrossEntropyLossLayer["Index"],
  BatchSize -> 64, MaxTrainingRounds -> 10, TargetDevice -> "GPU"
  ]
```

```
cm = ClassifierMeasurements[netT["TrainedNet"], X -> Y]
```

```
cm["ConfusionMatrixPlot"]
```

```
cm["Precision"]
```

> <|0->0.569919,1->0.580981|>

```
cm["Recall"]
```

> <|0->0.641173,1->0.506965|>

Adjusting the threshold

```
proba = netT["TrainedNet"][X, "Probabilities"];
proba0 = Lookup[proba, 0];
proba1 = Lookup[proba, 1];
```

```

PairedHistogram[Pick[proba0, Y, 0], Pick[proba0, Y, 1]]
```

```
Table[
  {
    Select[Pick[proba0, Y, 0], # >= i &] // Length,
    Select[Pick[proba0, Y, 1], # >= i &] // Length
    } // {i, ##, N[#[[1]]/(#[[1]] + #[[2]])]} &,
  {i, 0.5, 0.7, 0.01}
  ] // MatrixForm
```

```
PairedHistogram[Pick[proba1, Y, 1], Pick[proba1, Y, 0]]
```

```
Table[
  {
    Select[Pick[proba1, Y, 1], # >= i &] // Length,
    Select[Pick[proba1, Y, 0], # >= i &] // Length
    } // {i, ##, N[#[[1]]/(#[[1]] + #[[2]])]} &,
  {i, 0.5, 0.7, 0.01}
  ] // MatrixForm
```

Calculating the expected P&L

```
EV[p_] := p*80 - (1 - p)*100 - 4
```

```

Plot[
 {
   Select[Pick[proba0, Y, 0], # >= x &] // Length,
   Select[Pick[proba0, Y, 1], # >= x &] // Length
   } // EV[#[[1]]/(#[[1]] + #[[2]])]*(#[[1]] + #[[2]]) &,
 {x, 0.5, 0.7}
 ]
```

```
Plot[
 {
   Select[Pick[proba1, Y, 1], # >= x &] // Length,
   Select[Pick[proba1, Y, 0], # >= x &] // Length
   } // EV[#[[1]]/(#[[1]] + #[[2]])]*(#[[1]] + #[[2]]) &,
 {x, 0.5, 0.7}
 ]
```

## Answer by Jacques Joubert (score 4)

https://quant.stackexchange.com/a/44913

Whilst reading this I realized that it would be a really good application for meta-labeling. The idea behind meta-labeling is to build a secondary model that determines if the signals {0, 1} from the primary model are correct or not.

By doing this the secondary model outputs a value between 0 and 1 indicating how confident the model is that the primary model is correct or not. This output can then be passed to a bet sizing algorithm which maps the output to a position size. The core idea being that we want to take large positions on trades that are likely to be true and smaller positions on trades when we are unsure.

To give some intuition behind this. Lets take a trend following strategy as an example. Now moving average crossover strategies are known to under perform when the market moves sideways. The choppy nature causes a lot of transaction fees.

The secondary model will pick up that under some volatility conditions and perhaps a low auto correlation, that we are in a side ways trend and thus the primary models signal (a 1 in this case) is likely to be false and so it assigns it a low probability.

More about this technique can be read about in Chapter 3 of Advances in Financial Machine Learning. A toy example of meta-labeling can be found here Meta-Labeling on MNIST Data.