Four Lightweight Classification Models and Their Core Assumptions
Summary
This tutorial surveys support vector machines, k-nearest neighbors, naive Bayes, and the perceptron as lightweight classifiers for relatively small datasets and feature sets. It explains SVM’s maximum-margin boundary, the role of support vectors, soft margins for tolerating errors, and kernel mappings for nonlinear separation. KNN labels a sample from nearby training observations, using a chosen neighbor count and either equal or distance-weighted votes.
The naive Bayes section derives class probabilities from feature likelihoods and prior probabilities, emphasizing its conditional-independence assumption and the use of Laplace smoothing for unseen categories. It distinguishes frequency-based categorical modeling from Gaussian assumptions for numeric features. The perceptron is described as a weighted feature sum passed through an activation function. The article is conceptual rather than empirical: it provides no comparative accuracy results, and its suitability claims are not evaluated on trading data. It notes that SVM training may be slower and that kernel choice can affect underfitting or overfitting.
Key ideas
- SVM selects a separating boundary by maximizing the margin, with support vectors near the boundary shaping the decision.
- Soft-margin SVM permits classification errors, while kernels can represent nonlinear boundaries through higher-dimensional mappings.
- KNN predicts from the labels of nearby training samples, with neighbor count and distance weighting affecting the vote.
- Naive Bayes uses feature likelihoods and class priors while assuming conditional independence among features.
- Laplace smoothing helps naive Bayes handle feature values absent from training data.
- A perceptron applies an activation function to a weighted combination of input features.
Tags
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.