XGBoost Gradient-Boosted Trees: Objectives, Regularization, and Split Gain
Summary
This tutorial explains XGBoost through the broader framework of supervised learning and gradient-boosted decision trees. It introduces loss functions for regression and classification, then describes an ensemble as the sum of tree outputs. Training adds one tree at a time to improve the current prediction, using a second-order Taylor approximation of the loss based on per-observation gradients and Hessians.
The document derives how regularization on leaf count and leaf weights leads to optimal leaf scores and a gain formula for evaluating candidate splits. A greedy procedure selects feature thresholds with the greatest gain and avoids splits whose improvement does not outweigh the complexity penalty. These equations explain the method; the tutorial reports no trading experiments or empirical performance evidence. It is an introductory account of the algorithm, not a trading strategy, and explicitly cautions that its models are not intended as live-trading guidance.
Key ideas
- XGBoost builds an ensemble by adding trees sequentially to improve predictions from the existing model.
- A second-order approximation uses loss gradients and Hessians to score candidate tree structures.
- Regularization penalizes both the number of leaves and the size of their output weights.
- Optimal leaf weights and split gains can be calculated from aggregated gradients and Hessians.
- The greedy split procedure favors the candidate with the highest gain and can stop when complexity costs outweigh improvement.
Tags
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.