Baseline
Baselines are simple, fast models that serve as the minimum performance benchmark that any more complex model must beat to justify its existence — establishing the "floor" of useful predictive performance before investing weeks of engineering into sophisticated architectures, because if a $1M GPU-trained deep learning model only marginally outperforms a 5-line logistic regression, the complexity, cost, and maintenance burden of the deep learning approach is not justified.
What Are Baselines?
- Definition: The simplest reasonable model for a given task — one that requires minimal engineering effort and serves as the reference point against which all more complex models are compared.
- The Golden Rule: "If your fancy model can't beat the baseline, your fancy model is broken — or the problem doesn't need a fancy model."
- Why They Matter: Baselines reveal whether a problem is easy (baseline already achieves 95%), hard (baseline achieves 55%), or impossible with the given features (baseline achieves random chance). This information is critical before committing to complex approaches.
Standard Baselines by Task
| Task | Baseline | What It Does | Expected Performance |
|---|---|---|---|
| Binary Classification | Majority class predictor | Always predict the most common class | Accuracy = majority class % |
| Multi-class Classification | Most frequent class | Always predict the most common label | Accuracy = largest class % |
| Regression | Mean predictor | Always predict the training set mean | RMSE = standard deviation of target |
| Regression | Median predictor | Always predict the training set median | Robust to outliers |
| Time Series | Last value (persistence) | Tomorrow's value = today's value | Surprisingly strong for many series |
| Time Series | Moving average | Average of last N values | Simple smoothing |
| NLP Classification | TF-IDF + Logistic Regression | Bag of words + linear model | Often 80-90% of BERT performance |
| Recommendation | Most popular items | Recommend globally popular items | Strong for cold-start users |
| Object Detection | Sliding window + simple classifier | Exhaustive spatial search | Slow but functional |
The Baseline Ladder
| Level | Model | Engineering Effort | Purpose |
|---|---|---|---|
| 1. Trivial | Majority class / mean predictor | 1 line | Absolute floor |
| 2. Simple ML | Logistic Regression / Random Forest | 5-10 lines | "Is this problem learnable?" |
| 3. Strong ML | XGBoost with basic features | 20-50 lines | "How far can traditional ML go?" |
| 4. Deep Learning | BERT / ResNet / custom architecture | 100-1000+ lines | "Is the complexity justified?" |
Real-World Examples
| Problem | Trivial Baseline | Simple ML Baseline | Complex Model | Justified? |
|---|---|---|---|---|
| Spam detection | Always "not spam" (85%) | TF-IDF + LR (97%) | BERT (98%) | No — LR is good enough |
| Image classification | Random guess (10% on 10 classes) | HOG + SVM (75%) | ResNet (95%) | Yes — 20% improvement |
| Churn prediction | Always "not churn" (92%) | RF with basic features (88% F1) | XGBoost tuned (89% F1) | Marginal |
| Machine translation | Word-by-word dictionary | Statistical MT (BLEU 25) | Transformer (BLEU 45) | Yes — massive improvement |
Baselines are the essential first step of any machine learning project — establishing the minimum performance threshold that complex models must exceed to justify their cost, revealing whether the problem is genuinely solvable with the available data, and often demonstrating that simple models achieve 90% of the performance at 1% of the complexity.