Levenshtein Transformer

Levenshtein Transformer is a text generation model that generates and edits sequences using three edit operations: insertion, deletion, and replacement — inspired by the Levenshtein edit distance, the model iteratively transforms an initial (possibly empty) sequence into the target through a series of learned edit steps.

Levenshtein Transformer Operations

• Token Deletion: Predict which tokens to delete — a binary classification at each position.
• Placeholder Insertion: Predict where to insert new tokens — add placeholder positions for new tokens.
• Token Prediction: Fill in the placeholder positions with actual tokens — predict the inserted tokens.
• Iteration: Repeat deletion → insertion → prediction until convergence or a fixed number of steps.

Why It Matters

• Edit-Based: Natural for iterative refinement — the model can fix specific errors without regenerating the entire sequence.
• Adaptive Length: Unlike fixed-length NAT, the Levenshtein Transformer can dynamically adjust output length through insertions and deletions.
• Flexible Decoding: Can start from any initial sequence — including a rough draft, copied source, or empty sequence.

Levenshtein Transformer is text generation as editing — building and refining sequences through learned insertion, deletion, and replacement operations.