Naive Bayes
Naive Bayes is a family of fast, probabilistic classifiers based on Bayes' theorem that assume all features are conditionally independent given the class label — despite this "naive" assumption being almost never true in practice (words in an email are correlated, pixel values in an image are correlated), Naive Bayes works surprisingly well for text classification, spam filtering, and sentiment analysis, serving as the gold-standard baseline that more complex models must beat to justify their complexity.
What Is Naive Bayes?
- Definition: A generative classifier that uses Bayes' theorem — $P(Class|Features) = \frac{P(Features|Class) \times P(Class)}{P(Features)}$ — to calculate the probability of each class given the input features, then predicts the class with the highest probability.
- The "Naive" Assumption: All features are conditionally independent given the class. For spam detection, this means P("free" | Spam) is calculated independently of P("win" | Spam) — as if the presence of "free" tells you nothing about whether "win" also appears. This is obviously false (spam emails contain both), but the simplification makes computation tractable and the results are remarkably accurate.
- Why It Works Despite Being Wrong: The independence assumption affects the probability estimates but often preserves the ranking — if P(Spam|features) > P(Ham|features) with the naive assumption, it's usually true without it too.
Naive Bayes Variants
| Variant | Feature Type | Use Case | P(feature | class) Distribution |
|---|---|---|---|---|
| Multinomial NB | Word counts / frequencies | Text classification, spam filtering | Multinomial distribution | |
| Bernoulli NB | Binary (present/absent) | Short text, binary features | Bernoulli distribution | |
| Gaussian NB | Continuous (real-valued) | General classification, sensor data | Gaussian (normal) distribution | |
| Complement NB | Word counts (imbalanced) | Imbalanced text classification | Complement of each class |
Spam Classification Example
| Step | Process | Calculation | |
|---|---|---|---|
| 1. Prior | P(Spam) from training data | 30% of emails are spam → P(Spam) = 0.3 | |
| 2. Likelihood | P("free" | Spam) from word frequencies | "free" appears in 80% of spam → 0.8 |
| 3. Likelihood | P("meeting" | Spam) | "meeting" appears in 5% of spam → 0.05 |
| 4. Posterior | P(Spam | "free", "meeting") ∝ 0.3 × 0.8 × 0.05 | = 0.012 |
| 5. Compare | P(Ham | "free", "meeting") ∝ 0.7 × 0.1 × 0.6 | = 0.042 |
| 6. Decision | Ham wins (0.042 > 0.012) | Classify as Ham |
Strengths and Weaknesses
| Strength | Weakness |
|---|---|
| Extremely fast training (single pass through data) | Independence assumption is always violated |
| Works well with small datasets | Can't capture feature interactions |
| Handles high-dimensional data (10,000+ features) | Probability estimates are often poorly calibrated |
| Excellent baseline for text classification | Continuous features require distribution assumption |
| Scales linearly with data size | Outperformed by ensemble methods on tabular data |
When to Use Naive Bayes
- Text Classification: Spam filtering, sentiment analysis, topic categorization — Multinomial NB is often the first model to try.
- Baseline Model: Always train a Naive Bayes first. If a complex deep learning model only marginally beats it, the complexity isn't justified.
- Real-Time Systems: Sub-millisecond inference makes it suitable for high-throughput classification.
- Small Datasets: Still performs well with hundreds rather than millions of training examples.
Naive Bayes is the "unreasonably effective" baseline classifier — proving that a mathematically simple model with a provably wrong assumption can outperform complex algorithms on text classification tasks, and serving as the benchmark that every sophisticated model must justify its additional complexity against.