Frozen Features

Frozen Features refers to neural network representations that are not updated during training — the backbone weights are fixed (gradients not computed), and only the downstream task head is trained, preserving the original pre-trained feature space.

What Are Frozen Features?

• Mechanism: Set `requires_grad = False` for backbone parameters. Only the classification/regression head has gradients.
• Equivalence: Linear probing = frozen features + linear head. Feature extraction = frozen features + any downstream model.
• Storage: Features can be pre-computed and saved to disk for fast downstream experimentation.

Why It Matters

• Speed: Orders of magnitude faster training (no backprop through the backbone).
• Memory: Much lower GPU memory (no need to store intermediate activations for gradient computation).
• Fairness: Provides a standardized comparison by isolating the quality of the representation from the optimization procedure.

Frozen Features are the read-only mode of neural networks — locking down the learned representations to evaluate their intrinsic quality or enable efficient downstream adaptation.