Bayesian

Bayesian Deep Learning is the framework that treats neural network weights as probability distributions rather than fixed values — enabling principled uncertainty quantification by maintaining a posterior distribution over all possible model parameters, producing predictions that account for both aleatoric uncertainty in data and epistemic uncertainty from limited training.

What Is Bayesian Deep Learning?

• Definition: Apply Bayesian inference to neural networks — instead of finding a single optimal weight vector θ* via maximum likelihood, maintain a posterior distribution P(θ|data) over all possible weight configurations and integrate over this distribution to make predictions.
• Standard Deep Learning: θ* = argmax P(data|θ) — find single best weights, output single prediction.
• Bayesian Deep Learning: P(y|x, data) = ∫ P(y|x, θ) P(θ|data) dθ — average over all plausible weight configurations weighted by posterior probability.
• Core Challenge: For networks with millions of parameters, computing the true posterior is computationally intractable — requiring approximation methods.

Bayes' Rule Applied to Networks

P(θ|data) = P(data|θ) × P(θ) / P(data)

• Prior P(θ): Beliefs about weights before seeing data (typically Gaussian: weight regularization is a Gaussian prior).
• Likelihood P(data|θ): How well weights explain training data (cross-entropy loss is negative log-likelihood).
• Posterior P(θ|data): Updated beliefs about weights after seeing data — the target distribution.
• Marginal Likelihood P(data): Normalizing constant — computationally intractable for large networks.

Why Bayesian Deep Learning Matters

• Epistemic Uncertainty: The posterior spread over weights naturally represents the model's uncertainty about what the correct weights are — wide posterior = high epistemic uncertainty = model doesn't have enough data to be confident.
• Out-of-Distribution Detection: When test inputs fall outside the training distribution, the posterior predictive variance is high — the model correctly expresses uncertainty on novel inputs rather than outputting overconfident wrong answers.
• Active Learning: Epistemic uncertainty from the posterior identifies which unlabeled examples would most reduce posterior uncertainty — directing data collection efficiently.
• Catastrophic Forgetting: Bayesian methods like EWC (Elastic Weight Consolidation) use the Fisher information matrix (approximation of posterior curvature) to prevent overwriting important weights during continual learning.
• Scientific Applications: In physics, chemistry, and biology, Bayesian neural networks provide calibrated uncertainties for surrogate models — uncertainty estimates guide which expensive experiments to run next.

Approximation Methods

Variational Inference (Mean-Field):
- Approximate posterior P(θ|data) with a factored Gaussian Q(θ) = ∏ N(μ_i, σ_i²).
- Optimize ELBO (evidence lower bound): L = E_Q[log P(data|θ)] - KL(Q P(θ)).
- Results in "Bayes by Backprop" (Blundell et al.) — each weight has learnable mean and variance.
- Limitation: Mean-field assumption ignores weight correlations; underestimates posterior uncertainty.

Laplace Approximation:
- Train network normally to find θ* (MAP estimate).
- Fit a Gaussian at θ* using the Hessian of the loss: P(θ|data) ≈ N(θ*, H⁻¹).
- Modern approach (Daxberger et al.): Last-layer Laplace is computationally feasible for large networks.

Monte Carlo Dropout (Practical Gold Standard):
- Gal & Ghahramani (2016): Dropout training + dropout at inference = approximate Bayesian inference.
- Run T stochastic forward passes; mean = prediction; variance = uncertainty.
- No architecture change required — instant Bayesian uncertainty from any dropout-trained network.

Deep Ensembles:
- Train N networks from different random initializations.
- Lakshminarayanan et al. (2017): Ensembles are not Bayesian but empirically outperform most Bayesian approximations.
- Simple, parallelizable, and often the best practical uncertainty method.

Bayesian Deep Learning vs. Alternatives

Bayesian deep learning is the principled framework for uncertainty-aware neural networks — by maintaining distributions over weights rather than point estimates, Bayesian models genuinely know what they don't know, providing the epistemic foundation for trustworthy AI in scientific, medical, and safety-critical applications where confidence calibration is as important as prediction accuracy.