Mask-Predict

Mask-Predict is a non-autoregressive text generation strategy that iteratively predicts masked tokens — starting from a fully masked sequence, the model predicts all tokens simultaneously, then masks the least confident predictions and re-predicts them, repeating for a fixed number of iterations.

Mask-Predict Algorithm

• Initialize: Start with a fully masked sequence of predicted length N: [MASK] [MASK] ... [MASK].
• Predict: Generate all tokens simultaneously using a conditional masked language model.
• Mask: Mask the $k$ tokens with the lowest prediction confidence — $k$ decreases each iteration.
• Repeat: Re-predict the masked positions conditioned on the unmasked tokens — iterate T times (typically 4-10).

Why It Matters

• CMLM: Introduced by Ghazvininejad et al. (2019) for machine translation — dramatically faster than autoregressive decoding.
• Quality: 4-10 iterations achieve quality competitive with autoregressive translation — far fewer computation steps.
• Confidence-Based: Masking low-confidence tokens focuses computation where it's most needed — efficient refinement.

Mask-Predict is confident tokens stay, uncertain ones retry — iteratively improving generated text by re-predicting the least confident token positions.