Reinforcement Learning

Reinforcement Learning Fundamentals

RL Overview
Agent learns by interacting with environment, receiving rewards for good actions.

[Agent]
    |
    | action
    v
[Environment]
    |
    | state, reward
    v
[Agent] (update policy)

Key Concepts
| Concept | Description |
|---------|-------------|
| State (s) | Current environment observation |
| Action (a) | Agent choice |
| Reward (r) | Feedback signal |
| Policy (π) | Maps states to actions |
| Value (V) | Expected cumulative reward |
| Q-value (Q) | Expected reward for action in state |

RL Algorithms

Value-Based (Q-Learning, DQN)
Learn value of state-action pairs:

# Q-learning update
Q[s][a] = Q[s][a] + lr * (reward + gamma * max(Q[s_next]) - Q[s][a])

Policy Gradient
Directly optimize policy:

# Policy gradient update
loss = -log_prob(action) * advantage

Actor-Critic
Combine value estimation with policy optimization:

# Critic estimates value
value = critic(state)

# Actor updates policy using advantage
advantage = reward + gamma * critic(next_state) - value
actor_loss = -log_prob(action) * advantage

Common Algorithms
| Algorithm | Type | Use Case |
|-----------|------|----------|
| DQN | Value-based | Discrete actions |
| PPO | Policy gradient | General purpose |
| SAC | Actor-critic | Continuous control |
| A3C | Distributed | Parallel training |

RL for LLMs (RLHF)
Fine-tune LLMs with human preferences:

1. Collect human preference data
2. Train reward model
3. Use RL (PPO) to optimize LLM against reward

Libraries
| Library | Features |
|---------|----------|
| Stable Baselines3 | Ready-to-use algorithms |
| RLlib | Distributed RL |
| Gymnasium | Environments |
| Tianshou | Modular RL |

Challenges
| Challenge | Consideration |
|-----------|---------------|
| Sample efficiency | RL often needs many samples |
| Reward design | Reward hacking |
| Exploration | Balancing exploration vs exploitation |
| Stability | Training can be unstable |

Best Practices
- Start with well-tested algorithms (PPO)
- Normalize observations and rewards
- Monitor training closely
- Use domain knowledge for reward shaping
- Consider offline RL for data efficiency