Batch Rl

Batch RL is the original term for offline reinforcement learning — learning a policy from a fixed batch of previously collected transition data $(s, a, r, s')$ without any further interaction with the environment.

Batch RL Methods

• Fitted Q-Iteration (FQI): Iteratively fit the Q-function on the batch using supervised regression.
• LSPI: Least-Squares Policy Iteration — combine least-squares temporal difference with policy improvement.
• BCQ: Batch-Constrained Q-learning — only consider actions similar to those in the batch.
• BEAR: Bootstrapping Error Accumulation Reduction — constrain the policy's action distribution to match the data.

Why It Matters

• No Simulation Needed: Learn from real logged data — no simulator required.
• Extrapolation Error: The key challenge — the policy must not exploit Q-value errors for unseen state-action pairs.
• History: Batch RL predates the "offline RL" terminology — foundational work by Ernst et al., Lange et al.

Batch RL is the original offline RL — learning optimal policies from fixed datasets of previously collected transitions.