Insertion-Based Generation

Insertion-Based Generation is a text generation approach where the model builds the output sequence by inserting tokens into an initially empty (or seed) sequence — at each step, the model decides WHERE to insert and WHAT token to insert, growing the sequence from the inside out rather than left-to-right.

Insertion Generation Methods

• Balanced Binary Tree: Insert at the midpoint of gaps — $O(\log N)$ steps for a sequence of length $N$.
• Arbitrary Order: Learn to insert at any position — the model predicts both the position and the token simultaneously.
• Multiple Insertions: Insert multiple tokens per step — parallel insertion for faster generation.
• Stern-Brocot Tree: A specific insertion ordering that efficiently covers all positions.

Why It Matters

• Speed: $O(\log N)$ insertion steps vs. $O(N)$ for autoregressive — exponentially faster for long sequences.
• Bidirectional Context: Each inserted token can attend to BOTH left and right context — unlike left-to-right AR models.
• Flexibility: The generation order naturally adapts to the content — important words can be generated first.

Insertion-Based Generation is building text from the inside out — generating sequences by inserting tokens at chosen positions rather than strict left-to-right order.