Rl²

RL² (RL-Squared, Learning to Reinforcement Learn) is a meta-RL approach that uses a recurrent neural network to implement a learning algorithm within its activations — the RNN's hidden state acts as a learned RL algorithm, accumulating task-specific knowledge over the course of an episode.

How RL² Works

• Outer Loop: Train the RNN policy across many tasks via standard RL (this is the "meta" training).
• Inner Loop: At test time, the RNN adapts to a new task purely through its hidden state — no gradient updates.
• Input: The RNN receives $(s_t, a_{t-1}, r_{t-1}, d_{t-1})$ — state, previous action, reward, and done flag.
• Hidden State: The hidden state encodes the RNN's understanding of the current task — it IS the learned algorithm.

Why It Matters

• No Gradients at Test Time: Adaptation happens through forward passes — no backpropagation needed for new tasks.
• Learned Algorithm: The RNN can implement sophisticated exploration strategies (e.g., Thompson sampling emerges).
• Fast: Adaptation is as fast as a forward pass — real-time task adaptation.

RL² is a neural network that IS the RL algorithm — the RNN's hidden dynamics implement a learned reinforcement learning algorithm.