Generative Adversarial Imitation Learning

GAIL (Generative Adversarial Imitation Learning) is an imitation learning algorithm that uses a GAN-like framework to match the agent's state-action distribution to the expert's — a discriminator distinguishes expert from learner trajectories, and the learner's policy is trained to fool the discriminator.

GAIL Framework

• Discriminator: $D(s,a)$ — classifies whether $(s,a)$ came from the expert or the learner.
• Generator (Policy): $pi_\theta(a|s)$ — trained to produce behavior indistinguishable from the expert's.
• Reward: $r(s,a) = -\log(1 - D(s,a))$ — the discriminator's output serves as the RL reward.
• Training: Alternate between updating the discriminator (on expert vs. learner data) and the policy (using the discriminator reward).

Why It Matters

• No Reward Engineering: GAIL learns directly from demonstrations — no manual reward function design.
• Distribution Matching: Matches the entire occupancy measure, not just per-state actions — handles distribution shift.
• End-to-End: Combines IRL and RL into a single adversarial training loop — simpler than two-stage IRL.

GAIL is the GAN of imitation — adversarially matching the learner's behavior distribution to the expert's for robust imitation learning.