Born-Again Networks

Born-Again Networks (BAN) is a self-distillation technique where a model is re-trained using its own soft predictions as targets — the student has the identical architecture as the teacher, yet consistently outperforms the original teacher model.

How Do Born-Again Networks Work?

• Step 1: Train a teacher model normally with hard labels.
• Step 2: Train a student (same architecture) using the teacher's soft output distribution as the target.
• Step 3: Optionally repeat — use the student as the new teacher and train another generation.
• Result: Each generation improves, even with identical architecture.

Why It Matters

• Free Improvement: Same model, same data, better accuracy. The soft labels provide a richer training signal.
• Dark Knowledge: The teacher's soft outputs encode class-similarity information not present in hard labels.
• Sequence: Multiple generations of born-again training yield diminishing but consistent improvements.

Born-Again Networks are reincarnation for neural nets — proving that being trained on your own refined knowledge makes you smarter than your previous self.