Contextual Bandits
A contextual bandit is a reinforcement learning framework where an agent makes decisions based on context (features/state) available at decision time, receives a reward for its choice, but doesn't observe what would have happened with other choices. It sits between simple multi-armed bandits (no context) and full RL (sequential decisions).
How Contextual Bandits Work
- Observe Context: The agent receives a context vector $x$ — features describing the current situation.
- Select Action: Based on the context, the agent selects an action $a$ from a set of possible actions.
- Receive Reward: The environment returns a reward $r(x, a)$ for the chosen action.
- Learn: The agent updates its policy to improve future action selection.
- No observation of rewards for actions not taken (the counterfactual problem).
Examples
- News Recommendation: Context = user profile + time of day. Actions = articles to show. Reward = whether the user clicked.
- Ad Placement: Context = user demographics + page content. Actions = which ad to display. Reward = click or purchase.
- LLM Routing: Context = query characteristics. Actions = which model to send the query to. Reward = response quality score.
- Clinical Trials: Context = patient characteristics. Actions = treatment options. Reward = health outcome.
Key Algorithms
- LinUCB: Linear model for each action with Upper Confidence Bound exploration. Balances exploitation of known-good actions with exploration of uncertain ones.
- Thompson Sampling: Bayesian approach — maintain a posterior distribution over expected rewards for each action and sample from it to select actions.
- Epsilon-Greedy: With probability ε, explore randomly; otherwise, exploit the best-estimated action.
- Neural Contextual Bandits: Use neural networks to model the context-reward relationship for complex, high-dimensional contexts.
Contextual Bandits vs. Full RL
| Aspect | Contextual Bandit | Full RL |
|---|---|---|
| State | Single observation | Sequential states |
| Actions | One decision | Sequence of decisions |
| Consequence | Immediate reward | Delayed rewards |
| Complexity | Moderate | High |
Contextual bandits are the sweet spot for many real-world decision problems — they handle personalization and context while being simpler and more data-efficient than full reinforcement learning.