Sqil

SQIL (Soft Q Imitation Learning) combines imitation learning with soft Q-learning by treating expert demonstrations as receiving constant positive reward.

## What Is SQIL?

• Method: Assigns reward +1 to expert transitions, 0 to policy transitions
• Foundation: Built on Soft Actor-Critic (SAC) framework
• Simplicity: No adversarial training or reward learning required
• Sample Efficiency: Leverages off-policy replay from both expert and agent data

## Why SQIL Matters

SQIL achieves competitive imitation learning performance with minimal algorithmic complexity—just modify the reward signal in standard RL.

# SQIL reward assignment
def sqil_reward(transition, is_expert):
    if is_expert:
        return 1.0  # Expert demonstrations
    else:
        return 0.0  # Agent-generated transitions

# Training combines both buffers
expert_batch = sample(expert_buffer)
agent_batch = sample(agent_buffer)

# SAC update with SQIL rewards
sac_update(expert_batch, rewards=1.0)
sac_update(agent_batch, rewards=0.0)

Key Insight: The agent learns to stay close to expert states because deviating leads to zero reward, naturally encouraging imitation without explicit behavior cloning loss.