Calibrating Trading Probabilities Without Temporal Leakage
Summary
This article explains why classifier probabilities need calibration before they are used for trade sizing. A model can rank opportunities well while overstating their true win rates; that error can flow into probability-based sizing and Kelly-style adjustments, increasing exposure beyond what the evidence supports. Reliability diagrams visualize the difference between predicted confidence and observed outcomes, while Brier score, expected calibration error, and maximum calibration error assess complementary aspects of probability quality.
It compares isotonic regression, which flexibly maps raw probabilities to calibrated values but needs more data, with Platt scaling, which fits a logistic mapping and is more stable on smaller samples. To avoid temporal leakage, the proposed workflow fits calibration maps from out-of-fold predictions generated with PurgedKFold, then evaluates the full pipeline with combinatorial purged cross-validation. The article illustrates calibration effects and describes monitoring confidence intervals and recalibrating after regime or model changes. The method cannot make a weak or drifting model reliable; finite effective sample sizes and changing market distributions remain important limitations.
Key ideas
- A classifier can rank trades usefully while still producing probabilities that are systematically too high or too low.
- Probability errors can distort position sizing and Kelly-based exposure decisions.
- Reliability diagrams, Brier score, ECE, and MCE provide complementary calibration diagnostics.
- Isotonic regression is flexible but data-hungry, while Platt scaling uses a more constrained logistic mapping.
- Out-of-fold calibration with PurgedKFold helps avoid temporal leakage, but regime shifts can make a fitted calibrator stale.
Tags
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.