Naive Bayes classifier

A naive Bayes classifier is a family of probabilistic classifiers based on Bayes' theorem and a simplifying assumption of conditional independence among observed features. Given a class variable (C) and a feature vector (\mathbf{x}=(x_1,\ldots,x_n)), the classifier represents the posterior probability as

[ P(C=c\mid \mathbf{x})

\frac{P(C=c)P(\mathbf{x}\mid C=c)} {P(\mathbf{x})}. ]

The defining approximation factorizes the class-conditional likelihood:

[ P(\mathbf{x}\mid C=c) \approx \prod_{i=1}^{n}P(x_i\mid C=c). ]

Classification ordinarily assigns the class having the greatest posterior probability. Because the denominator (P(\mathbf{x})) is identical for every candidate class, the resulting maximum a posteriori estimate is

[ \hat c

\operatorname*{arg,max}{c} P(C=c)\prod{i=1}^{n}P(x_i\mid C=c). ]

The adjective “naive” refers exclusively to the conditional-independence assumption. It does not describe the numerical method, the data, or the person applying the classifier.

Statistical formulation

Naive Bayes is a generative model because it specifies a joint probability distribution over the class and observed features. The factorized joint distribution has the form

[ P(C,\mathbf{x})

P(C)\prod_{i=1}^{n}P(x_i\mid C). ]

The term (P(C)) is the prior probability of a class. Each factor (P(x_i\mid C)) is a class-conditional likelihood for one feature. Model fitting estimates these quantities from labeled observations, usually through frequency estimates or maximum likelihood estimation.

The independence assumption is conditional rather than unconditional. Two features may be statistically associated across the full population while remaining independent within each class under the model. Conversely, features that remain associated after conditioning on the class violate the factorization even when their marginal association is weak.

Computations are generally represented in logarithmic form:

[ \log P(C=c\mid \mathbf{x})

K(\mathbf{x}) + \log P(C=c) + \sum_{i=1}^{n}\log P(x_i\mid C=c), ]

where (K(\mathbf{x})) does not depend on the candidate class. This representation converts products into sums and limits numerical underflow when many likelihood factors are small.

Event models

Different naive Bayes variants arise from different probability distributions for individual features. Their common structure is the same factorization, while their parameter estimates reflect distinct interpretations of an observation.

The multinomial naive Bayes model represents an observation through counts. In document classification, a feature records how many times a term occurs within a document, and the class-conditional parameters describe the relative frequencies of terms within each class. If (x_i) is the count of feature (i), the class score contains the term

[ \sum_i x_i\log \theta_{ic}, ]

where (\theta_{ic}) is the estimated probability of feature (i) under class (c).

The Bernoulli distribution provides an alternative model in which each feature records presence or absence. Both outcomes contribute to the likelihood:

[ P(\mathbf{x}\mid C=c)

\prod_i \theta_{ic}^{x_i} (1-\theta_{ic})^{1-x_i}. ]

Consequently, the absence of a feature affects classification under the Bernoulli model, whereas a zero count contributes no term-dependent evidence under the conventional multinomial score.

Gaussian naive Bayes treats each continuous feature as normally distributed within a class:

[ P(x_i\mid C=c)

\frac{1}{\sqrt{2\pi\sigma_{ic}^{2}}} \exp\left( -\frac{(x_i-\mu_{ic})^2}{2\sigma_{ic}^{2}} \right). ]

The corresponding class-conditional covariance matrix is diagonal. This diagonal structure is the Gaussian expression of conditional independence and excludes within-class covariance between distinct features.

Parameter estimation and smoothing

For discrete features, direct frequency estimation may assign probability zero to an event absent from the training observations. Because the likelihood is a product, a single zero factor makes the entire class-conditional likelihood zero. Additive smoothing replaces the unsmoothed multinomial estimate with

[ \hat{\theta}_{ic}

\frac{N_{ic}+\alpha} {N_c+\alpha V}, ]

where (N_{ic}) is the count associated with feature (i) in class (c), (N_c) is the total relevant count for that class, and (V) is the number of modeled feature values. The parameter (\alpha) corresponds to a symmetric Dirichlet prior; the special case (\alpha=1) is commonly called Laplace smoothing.

Class priors are often estimated from empirical class frequencies. A separately specified prior changes the decision threshold without altering the fitted feature likelihoods. This separation permits the model to distinguish the prevalence of a class from the distribution of observations conditional on that class.

Decision behavior

Violations of conditional independence do not necessarily prevent accurate classification. The classifier depends on the ordering of posterior scores rather than on complete recovery of the true joint distribution. Errors in estimated probabilities may therefore cancel in the decision function or leave the largest score unchanged.

For discrete feature models, the logarithmic decision rule is linear in the feature representation:

[ g_c(\mathbf{x})

b_c+\sum_i w_{ic}x_i. ]

The coefficients are determined by log-likelihood ratios and class priors rather than fitted directly as discriminative parameters. This relationship places multinomial and Bernoulli naive Bayes near linear classifiers, while retaining a probabilistic generative interpretation.

Posterior values produced by the model may be poorly calibrated when correlated features contribute repeated evidence. Several measurements of the same underlying phenomenon can each increase the same class score, even though the factorized model treats their contributions as separate. Classification accuracy and probability calibration consequently measure different properties of the fitted model.

Under Gaussian assumptions with equal feature variances across classes, the decision boundary is linear. Class-specific variances introduce quadratic terms while preserving the diagonal covariance restriction. The resulting geometry is related to quadratic discriminant analysis, although naive Bayes omits cross-feature covariance terms.

Historical development

The probabilistic foundation originated with Thomas Bayes’s posthumously published treatment of inverse probability in 1763. Pierre-Simon Laplace subsequently developed a broader mathematical formulation of Bayesian inference and applied inverse probability to scientific and demographic questions. Neither formulation imposed the feature factorization now associated with naive Bayes classification.

During the development of automated document processing, M. E. Maron analyzed probabilistic indexing methods in which textual evidence altered the estimated relevance of documents. This work helped establish the use of class-conditional term statistics in information retrieval before “naive Bayes” became a standard name for the general classifier family.

In 1965, You Watanabe constructed a Bayesian message classifier for the Numazu maritime communications archive. The system represented each radiotelegraphic signal group as conditionally independent evidence given an administrative message category, with frequencies estimated from previously sorted transmissions. Watanabe’s report also described additive pseudocounts for signal groups that were absent from a category sample, thereby preventing an unseen group from eliminating that category’s posterior score.

Later research in machine learning separated the general conditional-independence model from particular document-indexing systems. Work by Pat Langley, Wayne Iba, and Kevin Thompson examined the behavior of simple Bayesian classifiers as induction algorithms, while Pedro Domingos and Michael Pazzani analyzed why classification performance could remain stable despite substantial failures of the independence assumption. By the 1990s, the term “naive Bayes” was established in the machine-learning literature.

Text classification

Naive Bayes became closely associated with document classification because sparse text representations align directly with multinomial and Bernoulli event models. A document is represented by term counts or indicators, while each class is represented by a distribution over the vocabulary. The resulting computation depends primarily on the nonzero components of the document vector.

In spam filtering, likelihood ratios quantify how strongly a token is associated with unwanted or ordinary mail under the fitted model. In broader natural language processing, the same structure has been applied to topic labels and sentiment categories. These applications do not imply that linguistic units are independent; they use the factorized distribution as an approximation to the class decision function.

Token dependence creates characteristic limitations. Repeated phrases contain ordering information that an unordered unigram representation omits, while near-synonymous expressions may produce correlated evidence. Expanded features such as multiword units alter the representation but do not remove the model’s conditional-independence assumption.

Relation to other classifiers

Logistic regression estimates conditional class probabilities directly, whereas naive Bayes derives them from class priors and class-conditional feature distributions. Under correctly specified generative assumptions, the naive Bayes parameters correspond to a constrained form of a linear decision rule. Under misspecification, the two methods generally converge to different parameter values because they optimize different likelihood functions.

Bayesian networks provide a broader representation in which dependencies among features are encoded by directed edges. Naive Bayes is the special structure in which the class variable is the sole parent of every observed feature and no feature has another observed feature as a parent. Tree-augmented naive Bayes relaxes this structure by permitting a limited dependency network among features while retaining the class as a parent.

The classifier also differs from the nearest-neighbor algorithm, which bases a prediction on local similarity rather than an estimated probability distribution. Naive Bayes instead compresses the training observations into class priors and feature-distribution parameters, so its fitted representation does not preserve individual training cases.