Backpropagation Through Time

Backpropagation Through Time (BPTT) is the standard algorithm for computing gradients in recurrent neural networks — unrolling the recurrent computation through time steps and applying the chain rule to propagate error gradients backward through the entire sequence.

How BPTT Works

• Unrolling: Unfold the RNN recurrence into a feedforward computation graph over $T$ time steps.
• Forward Pass: Compute all hidden states $h_1, h_2, \ldots, h_T$ and the loss $L$.
• Backward Pass: Apply the chain rule backward through all time steps to compute $\partial L / \partial \theta$.
• Weight Sharing: Gradients from all time steps are accumulated for the shared weight parameters.

Why It Matters

• Standard Method: BPTT is how all RNNs, LSTMs, and GRUs are trained.
• Vanishing Gradients: Gradients can vanish or explode over long sequences — motivating LSTM and gradient clipping.
• Truncated BPTT: Practical variant that limits backpropagation to a fixed window for memory and stability.

BPTT is the chain rule unrolled through time — the fundamental algorithm for training sequence models by propagating gradients through temporal computation.