Denoising Objective

Denoising Objective is a general class of self-supervised learning objectives where the model is trained to reconstruct a clean input from a corrupted (noisy) version — fundamental to BERT (MLM), BART, T5, and Denoising Autoencoders, teaching the model the data distribution by learning to remove noise.

Common Corruptions (Noise)

• Masking: Hiding tokens ([MASK]).
• Deletion: Removing tokens.
• Infilling: Replacing spans with a single mask.
• Permutation: Shuffling order.
• Rotation: Rolling the sequence.
• Replacement: Swapping tokens with random ones.

The Goal

• Loss: Minimize reconstruction error (Cross-Entropy) between generated/predicted output and original clean input.
• Manifold Learning: By mapping noisy points back to data points, the model learns the "manifold" of structured language.
• Context Dependence: To fix noise, the model must understand the context — syntax, semantics, and facts.

Denoising Objective is learning by fixing — the core principle of modern NLP pre-training: corrupt the data and teach the model to repair it.