Evaluating Model Fit and Selecting Features with Greedy Search

Summary

This tutorial introduces bias, variance, and irreducible error as sources of prediction error. It relates model complexity to underfitting and overfitting: overly simple models tend to perform poorly on both training and test data, while overly complex models can fit training data well but generalize poorly. The goal is to choose complexity that reduces overall error, assessed through out-of-sample prediction or model criteria such as AIC and BIC.

It then explains why feature selection can remove irrelevant or redundant inputs and limit complexity. Exhaustively evaluating every subset becomes expensive as the feature set grows, so the tutorial describes forward selection, backward elimination, and bidirectional search. These greedy methods add or remove features based on whether a chosen score improves. They reduce computation, but may miss the globally best subset. The examples are conceptual; the document does not provide empirical trading results, and its code implementation is announced but not included in the supplied text.

Key ideas

• Prediction error can reflect bias, variance, and uncertainty in the data itself.
• Underfitting is associated with poor training and test performance, while overfitting can impair test performance despite a good training fit.
• Feature selection can reduce irrelevant or redundant inputs and help control model complexity.
• Exhaustive feature subset evaluation grows rapidly as the number of candidate features increases.
• Forward, backward, and bidirectional greedy searches save computation but may miss the best possible subset.
• Feature combinations can be scored with test performance or criteria such as AIC and BIC.

Tags

This summary was written by Stratmill's research agent from the original; it is not a copy of the source.