Detecting Leakage in CUSUM-Filtered XGBoost Trading Models
Article Quant Q&A · Author: Federico Juvara
Summary
The author describes an hourly forex classification pipeline using fractional differentiation, CUSUM event sampling, technical features, and XGBoost. Reported test accuracy rises substantially in some configurations, especially when labels are based on changes in fractionally differentiated prices or when CUSUM filtering is retained. Yet the live model often repeats one direction, and removing particular preprocessing steps reduces reported accuracy toward chance levels.
Replies raise leakage and label construction as central concerns. They point to label overlap and suggest weighting labels and using sequential bootstrapping, as well as considering meta-labeling. Another response questions the usefulness of differentiating prices for labels and proposes checking whether results persist as the CUSUM threshold approaches zero. The thread does not establish the exact source of bias or demonstrate profitability; its reported accuracy is not proof of a valid out-of-sample strategy, and the implementation details are insufficient for a definitive diagnosis.
Key ideas
- Applying event filters and transformations across the full dataset can create leakage risks.
- High classification accuracy does not establish that a trading strategy is profitable or unbiased.
- Overlapping labels may require careful weighting and sampling methods.
- Compare label definitions and preprocessing choices with strictly time-ordered validation.
- The discussion suggests threshold sensitivity checks for diagnosing possible CUSUM leakage.
Tags
Full text
# Leakage and bias in XGBoost trading strategy # Leakage and bias in XGBoost trading strategy I apologize for my persistence, i'm on a course of study and doubts increase every day. My goal is "just" to code a profitable forex trading strategy with machine learning. I'm trying to implement de Prado's techniques with time bars (ticks are expensive and require strong hardware) but it seems a mess. These are the crucial steps: - Load data, in my case Fxcm or csv free data. Granularity 1 hour. - Cleaning, missing values, datetime index correction, rename ecc. - Dollar value calculation (with volume or tickqty series) - Dollar value filtering using "compute dollar bars" from Lopez de Prado's book (this step was eliminated, it doesn't improves accuracy and seems incorrect outside tick data) - Fractional Differentiation (0.2/0.5 range) of the entire dataframe, using getDailyVol and getTEvents from Lopez de Prado's book. - Application of the CUSUM filter to undersample the dataset selecting the most relevant samples (This affects the entire dataset, not just train, so here i have a feeling of a great bias) - A custom function creates needed features: return (from raw close series, not yet frac differentiated), Ewm, rolling means, std, ratios, volatility, technical indicators of all sorts ecc. - The latter function creates target labels: return shifted by negative one for regression tasks, return shifted by negative one converted in binary labels for classification tasks (if next hour's return is positive 1, else -1) - Train/test split of my dataset: X = all my features except the last two, y = values. reshape of one of the above mentioned features - XGBoost model implementation: StratifiedKFold on X_train and y_train + validation, hyper parameter tuning divided in 7/8 steps to reduce computational impact. Best params used to fit the model on X_train and y_train, then prediction on X_test. - Save model in pickle, load model with pickle in another Jupyter file This file streams and prints via fxcmpy API the last hourly candles available. A bunch of custom functions do what follows: every x minutes check if a new candle was updated. If no do recheck, if yes activate a function that do what follows: apply the entire preprocessing step mentioned above to last x available candles. I mean frac diff, technical indicators, returns ecc. I provide to my model the same features iwith which it was trained. I bypass only dollar value filtering and Cusum. After that make prediction using last candle. If the signal says 1 go long, if it says -1 go short. Two variables calculate risk and position sizing risk asjusted by predict_proba function and a stop loss predefined (let's say 10 pips). Exit with next updated candle or after a 1:1,5 risk reward ratio is reached. If next signal has the same value and no target is reached hold the position. My broker send orders. This model get an accuracy percentage of 70/75% in test data, but going live i have low sensitivity, direction remains the same for many bars, regardless of predict_proba that may vary. For example taking 10 bars model will predict 9/10 buy signals. Data and value_counts are not unbalanced . My results are the following (i'm talking about model training, let's forget about live bot): - by eliminating CUSUM filter and frac diff just using raw returns as binary label accuracy drops to a random percentage, maybe 50/52%. - by eliminating CUSUM filter but keeping frac diff i get same results. - by eliminating CUSUM filter keeping frac diff and using as binary label fractionally differentiated close series accuracy rockets, 70/75%. Just to be clear, label code takes the following form: df['yfrac'] = np.where(df['frac_diff_close'].shift(-1) > df['frac_diff_close'], 1, -1) - keeping CUSUM filter and frac diff and using raw return as binary label accuracy rockets as well. - keeping CUSUM filter and frac diff and using fractionally differentiated raw returns accuracy obtains the best result (don't ask me why) So what differentiates a trash random strategy from a potentially profitable one are CUSUM filter and frac differentiated binary labels. Don't ask me why. Does it makes sense to use a frac diff close series as binary label and CUSUM filter to gain accuracy without leakage? CUSUM filter in test data is a look-ahead bias? ## Answer by Payam Skandari (score 1) https://quant.stackexchange.com/a/66532 The main issue seems to be spillage, according to De Prado. Labels need to be assigned proper weights and picked in bagging process using sequential bootstrapping. The other important point which seems to be overlooked in the list of steps you outlined is the Meta-labeling process. I did not get good results before I applied this method. ## Answer by mr mr (score 0) https://quant.stackexchange.com/a/77965 You dont need to do the frac diff of the price changes: I assume it gives exactly the same results as if you use the original price serices. df['yfrac'] = np.where(df['frac_diff_close'].shift(-1) > df['frac_diff_close'], 1, -1) It's pretty clear you have data leakage in your cusum filter/frac diff implementation. You could try to reduce the "threshold" in the Cusum filter to zero, if you still see a difference in the accuracy, then it's certain you have data leakage in your cusum implementation. ## Answer by Adrian Velai (score -2) https://quant.stackexchange.com/a/85881 Check at https://www.bitcoinai.pro/data-leakage-financial-ml/ how I solved the leak; maybe it helps. A practical checklist anti leakage Order every split chronologically; never shuffle bars. Purge every boundary by more than your maximum holding period. Add an embargo beyond the purge for long-lookback features. Compute features causally — no centred windows, no forward-filled fundamentals, no survivorship-filtered universes. Model costs at the venue’s real commissions and mask structural artifacts rather than exploiting them. Write the promotion rule before the search runs. Rebuild every published number from its frozen artifact before publishing it. Make protections fail loudly; a silently disabled safeguard is worse than no safeguard, because it buys false confidence.
Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.