Feature Scaling in Gradient Tree Boosting
Summary
This discussion asks whether input attributes need to be scaled before gradient tree boosting. One answer says scaling is generally unnecessary for tree-based boosting, since the distance-based concerns that motivate normalization for methods such as support vector machines do not apply in the same way. It recommends comparing performance with and without scaling when the preprocessing cost is low, and skipping it when the cost is high.
A second answer favors scaling for some classification settings and suggests considering decorrelation, but it also recommends trying alternatives because boosting methods and feature sets differ. It lists transformations such as reducing skew, standardizing, and mapping values to a fixed range. These recommendations are not supported by a shared benchmark in the discussion, and the broad claims about ensembles and feature selection are opinions rather than demonstrated results. The practical lesson is to match preprocessing to the algorithm and task, then evaluate it empirically instead of assuming that scaling always helps or always harms.
Key ideas
- Gradient tree boosting generally does not require feature scaling for the reasons that motivate it in distance-based algorithms.
- Scaling can still be tested empirically when preprocessing is inexpensive.
- Different transformations include standardization and mapping features to a fixed range.
- Some practitioners recommend scaling or decorrelating features, but the discussion presents no comparative results.
- Preprocessing choices should be evaluated for the specific model and task.
Tags
Full text
# Gradient tree boosting -- do input attributes need to be scaled? # Gradient tree boosting -- do input attributes need to be scaled? For other algorithms (like support vector machines), it is recommended that input attributes are scaled in some way (for example put everything on a [0,1] scale). I have googled extensively and can't find any information on whether this needs to be done for boosting methods, and in particular gradient tree boosting. ## Answer by Theja Tulabandhula (score 9) https://quant.stackexchange.com/a/9195 > do input attributes need to be scaled? No. It is not required. It is only a heuristic [1]. It is primarily motivated because of the following: From the Feature Scaling article: - From the Feature Scaling article: > Since the range of values of raw data varies widely, in some machine learning algorithms, objective functions will not work properly without normalization. For example, the majority of classifiers calculate the distance between two points by the distance. If one of the features has a broad range of values, the distance will be governed by this particular feature. Therefore, the range of all features should be normalized so that each feature contributes approximately proportionately to the final distance. In summary, - The recommendation for other algorithms like SVM is just 'recommendation'. It does not guarantee improved performance for instance. - My suggestion is if this step is expensive, skip it. If it is not, then check to see if normalization does not deteriorate performance compared to building a Gradient boosted tree directly and decide accordingly. ## Answer by user6430 (score 3) https://quant.stackexchange.com/a/9500 For class prediction (vs. class discovery), I always transform (remove scale effects) for input features. Metrics for features with larger scale (range) can swamp metrics of features with smaller scale. Depending on your method (there are numerous boosting approaches), you might also consider decorrelating features. A golden rule regarding correlated features, at least with artificial neural networks, is that during connection weight learning time will be spent on figuring out the correlation -- herein lies the reason why many use linear dimensional reduction (e.g., PCA) prior to clamping objects to the input network nodes. Overall, you should probably do both (with and without). And recall, there are choices for scaling: (a) you can remove skewness (fat-tails), (b) mean-zero standardize, and (c) normalize in range [0,1], (d) determine direction cosines of the input features so feature values are within a unit circle. Last, never use a single classifier unless you need to. If possible, use an ensemble of classifiers so that one classifier can pick up on problematic objects when another breaks down. The No Free Lunch theorem states that in the universe of all cost functions, no one classifier is better than another - so never assume you have the best solution when using a single classifier. The Ugly Ducking theorem states that no one set of features is better than another -- so never assume your selected feature set is the best.
Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.