Conservative Q-Learning

CQL (Conservative Q-Learning) is the leading offline RL algorithm that learns a conservative (lower-bound) Q-function — penalizing Q-values for out-of-distribution actions to prevent the overestimation that causes standard Q-learning to fail in the offline setting.

CQL Algorithm

• Conservative Penalty: Minimize Q-values under the current policy: $\min_Q \alpha \mathbb{E}_{a \sim pi}[Q(s,a)]$.
• Data Support: Maximize Q-values for actions in the dataset: $\max_Q \mathbb{E}_{a \sim D}[Q(s,a)]$.
• Combined: $L_{CQL} = \alpha (\mathbb{E}_pi[Q] - \mathbb{E}_D[Q]) + L_{TD}$ — pushes down OOD, pushes up in-distribution.
• Lower Bound: The learned Q-function is a provable lower bound of the true Q-function — conservative but safe.

Why It Matters

• No Overestimation: CQL prevents the catastrophic overestimation of out-of-distribution actions that kills standard offline Q-learning.
• Provable: CQL provides provable lower bounds on the policy's true value — safe deployment guarantees.
• State-of-Art: CQL is one of the strongest offline RL baselines — works across many domains.

CQL is pessimistic Q-learning — learning a deliberately conservative value function for safe, reliable offline RL.