Dagger

DAgger (Dataset Aggregation) is an imitation learning algorithm that addresses behavioral cloning's distribution shift problem — iteratively collecting new expert labels for the states the LEARNER visits, aggregating them into the training dataset, and retraining the policy.

DAgger Algorithm

• Step 1: Train initial policy $pi_1$ via behavioral cloning on expert demonstrations $D$.
• Step 2: Roll out $pi_i$ to collect states visited by the learner.
• Step 3: Query the expert for the correct actions at these states — get expert labels for learner-visited states.
• Step 4: Aggregate: $D \leftarrow D cup D_{new}$, retrain $pi_{i+1}$. Repeat.

Why It Matters

• Distribution Shift Fix: By training on states the LEARNER visits (not just expert states), DAgger eliminates distribution shift.
• Theoretical: DAgger provides no-regret guarantees — the learned policy converges to expert performance.
• Interactive: Requires an interactive expert who can label learner states — not always available.

DAgger is learning from your own mistakes — iteratively getting expert feedback on the states the learner actually visits.