Gap Sentence Generation

Gap Sentence Generation (GSG) is a pre-training objective used in PEGASUS specifically designed for abstractive summarization — whole sentences are masked (removed) from a document, and the model (seq2seq) must generate these missing sentences.

Mechanism

• Selection: Select "important" sentences (e.g., using ROUGE scores vs. the rest of the doc) to act as pseudo-summaries.
• Masking: Remove these sentences from the input using [MASK1].
• Generation: The decoder must generate the exact text of the missing sentences.

Why It Matters

• Summarization Bias: Standard MLM doesn't teach summarization. GSG forces the model to synthesize content from the rest of the document.
• PEGASUS: Showed that this objective beats standard BERT/Roberta approaches on summarization (CNN/DailyMail, XSum) with far less data.
• Principle: Pre-training objectives should mimic the downstream task.

Gap Sentence Generation is summarization pre-training — forcing the model to generate key missing sentences, simulating the abstractive summarization process.