Confusion Matrix
Confusion Matrix is the fundamental evaluation tool for classification models that breaks accuracy into four categories — True Positives, True Negatives, False Positives, and False Negatives — revealing not just how often the model is right, but HOW it fails (does it miss actual positives or cry wolf too often?), enabling practitioners to optimize for the specific type of error that matters in their domain: cancer screening demands high recall (catch every case), spam filtering demands high precision (don't delete real email).
The Four Quadrants
| Predicted Positive | Predicted Negative | |
|---|---|---|
| Actually Positive | True Positive (TP) ✓ | False Negative (FN) — MISS |
| Actually Negative | False Positive (FP) — FALSE ALARM | True Negative (TN) ✓ |
Derived Metrics
| Metric | Formula | Question It Answers | Optimize When |
|---|---|---|---|
| Accuracy | (TP + TN) / Total | "How often is the model correct overall?" | Classes are balanced |
| Precision | TP / (TP + FP) | "Of all positive predictions, how many were right?" | False alarms are costly (spam filter) |
| Recall (Sensitivity) | TP / (TP + FN) | "Of all actual positives, how many did we catch?" | Missing positives is dangerous (cancer) |
| F1 Score | 2 × (Precision × Recall) / (Precision + Recall) | "What is the harmonic mean of precision and recall?" | You need both precision and recall |
| Specificity | TN / (TN + FP) | "Of all actual negatives, how many did we correctly identify?" | False positives are costly (drug testing) |
Real-World Trade-offs
| Domain | Priority | Why | Tolerance |
|---|---|---|---|
| Cancer Screening | High Recall (>95%) | Missing a cancer case can be fatal | Accept some false alarms (further testing is cheap) |
| Spam Filter | High Precision (>99%) | Deleting a real email is worse than letting spam through | Accept some spam in inbox |
| Fraud Detection | High Recall (~90%) | Missing fraud costs money | Accept investigating some legitimate transactions |
| Self-Driving Cars | High Recall for obstacles | Missing a pedestrian is catastrophic | Accept some false braking |
| Criminal Justice | High Precision | Wrongly convicting an innocent person is devastating | Accept some guilty going free |
Why Accuracy Is Misleading
With 99% healthy patients and 1% sick:
- A model that always predicts "Healthy" gets 99% accuracy but catches 0% of sick patients (Recall = 0).
- Accuracy masks complete failure on the minority class — always check precision and recall for each class separately.
The Precision-Recall Trade-off
- Increase threshold (require higher confidence for positive prediction) → Precision ↑ Recall ↓ (fewer but more confident positive predictions).
- Decrease threshold (accept lower confidence) → Precision ↓ Recall ↑ (catch more positives but with more false alarms).
- The F1 Score balances both — but domain requirements should determine which metric matters more.
Confusion Matrix is the essential diagnostic tool for classification models — revealing the specific failure modes that a single accuracy number hides, enabling practitioners to choose the right metric for their domain (precision vs recall), tune decision thresholds accordingly, and build models that fail in the least harmful way for their specific application.