Calibrating Bet Probabilities for News-Based Trading Models

Summary

The document describes a challenge in training a model to classify stock news as a long, short, or no-bet opportunity while producing usable probabilities. Expected-value optimization can favor taking bets even when the estimated edge is tiny, because occasional paths may reach both hypothetical profit targets without triggering stops. Log-profit utility penalizes risk near zero expected value, but the author reports that small changes in bet size can cause the model to switch abruptly between betting and abstaining.

The post asks how to make the model bet only when confident, but offers no solution, experiment, or performance evidence. It illustrates that the choice of objective and bet sizing can strongly affect probability outputs and trading decisions. The discussion is limited to the author's setup; it does not establish a general calibration method or compare alternatives such as explicit confidence thresholds or probability calibration procedures.

Key ideas

• Expected-value loss may encourage bets even when the estimated edge is very small.
• Log-profit utility can penalize risky bets with near-zero expected value.
• The model's decisions appear highly sensitive to bet size under the utility objective.
• The document raises, but does not answer, how to tie confidence to betting decisions.

Tags

# Having a hard time getting sentiment analysis bot to calibrate bet probabilities well


# Having a hard time getting sentiment analysis bot to calibrate bet probabilities well












Training a language model for taking bets in the stock market based on news. For each news release I have three categories for each stock: long bet, short bet, or neither. The NN outputs probabilities for each category.

I'm having a surprisingly hard time getting a good calibration for the probabilities. If I use EV as the loss function, there's no reason to ever not take a bet, no matter how small your opinion over the signal is, because you can set

P(long) = 0.5, P(short) = 0.5

for a very slightly positive EV (because once in a while the price hits both hypothetical profit takers without hitting the stop losses).

On the other hand I use expected utility as the loss, with utility = log(profit). This way a bet with EV close to 0 and positive risk has negative expected utility. However, as I tweak the size of the bet, the NN goes straight from ignoring the risk to never betting with very slight changes in bet size.

I could hunt for the perfect value, but I don't like how fragile results are to a hyperparameter choice.

How do I get the model to only bet when it's confident?

Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)

This summary was written by Stratmill's research agent from the original; it is not a copy of the source.