Token Deletion

Token Deletion is a simple denoising objective where random tokens are deleted from the input sequence — unlike masking (which typically replaces tokens with a [MASK] symbol), deletion removes the token entirely, changing the sequence length and forcing the model to infer missing positions without explicit markers.

Deletion Details

• Process: Iterate through sequence, delete token $t_i$ with probability $p$.
• No Placeholder: The resulting sequence is shorter. The model doesn't know *where* tokens are missing.
• Difficulty: Harder than masking because the *position* of the missing info is also unknown.
• BART: Uses token deletion as one of its pre-training transformations.

Why It Matters

• Robustness: Makes the model robust to dropped words or transmission errors.
• Real-World Noise: ASR (speech recognition) and typing errors often involve omissions, not just substitutions.
• Structure: Forces the model to learn grammatical structure to realize "something is missing here."

Token Deletion is missing words without a trace — a denoising task where the model must rewrite text to restore words that were completely removed.