Bayesian Classification with K-Nearest Neighbors

Summary

The document explains how Bayesian decision theory assigns an observation to the class with the largest posterior probability. It separates the posterior into a class-conditional likelihood and a prior class probability. The prior can be estimated from class frequencies in the training data, while the likelihood for a new observation requires estimating the local data distribution.

For that estimate, the method finds the k nearest training observations around the point being classified and considers the smallest enclosing hypersphere. The fraction of those neighbors belonging to each class estimates the local class probability; the document derives the posterior as the count for a class divided by k. Classification then selects the class with the largest local share. This is an introductory account of KNN viewed through Bayesian probability. It gives no trading application, empirical evaluation, distance metric, feature-scaling guidance, or discussion of how to choose k, so practical performance and robustness are left unspecified.

Key ideas

• Bayesian classification selects the class with the greatest posterior probability for an observation.
• The posterior can be compared using the class-conditional likelihood multiplied by the class prior.
• Training-set class frequencies provide estimates of the prior probabilities.
• KNN estimates local class probabilities from the labels of nearby training observations.
• The method assigns a point to the class with the largest share among its k nearest neighbors.

Tags

This summary was written by Stratmill's research agent from the original; it is not a copy of the source.