Temporal Ensembling

Temporal Ensembling is a semi-supervised learning method that maintains an exponential moving average of each sample's prediction over training epochs — using these accumulated predictions as soft targets for a consistency loss on unlabeled data.

How Does Temporal Ensembling Work?

• Accumulate: After each epoch, update the EMA prediction for each sample: $\tilde{z}_i = \alpha \tilde{z}_i + (1-\alpha) z_i$.
• Target: Use the corrected EMA $\hat{z}_i = \tilde{z}_i / (1 - \alpha^t)$ as the target.
• Consistency Loss: $\mathcal{L} = z_i - \hat{z}_i ^2$ (current prediction should match accumulated predictions).
• Paper: Laine & Aila (2017).

Why It Matters

• Ensemble Effect: The EMA predictions implicitly ensemble the model across training epochs.
• Simple: No additional model or parameters (just a prediction buffer).
• Limitation: Targets update only once per epoch (slow). Mean Teacher addresses this.

Temporal Ensembling is memory of past predictions — using the accumulated history of a sample's predictions as a stable learning target.