Summary Generation as Pre-Training

Summary Generation as Pre-training (or Gap Sentence Generation) is a pre-training strategy where the model learns to generate a summary of the input text — either using naturally occurring summaries (headlines, abstracts) or pseudo-summaries created by identifying key sentences in the document (PEGASUS).

Data Sources

• PEGASUS (GSG): Mask important sentences (those with high ROUGE overlap with the rest) and generate them.
• News Headlines: Predict the headline from the article body.
• Abstracts: Predict the abstract from the paper body.
• Reddit: Predict the post title or TL;DR from the body.

Why It Matters

• Abstraction: Forces the model to synthesize information, not just copy it.
• Importance Ranking: To summarize, the model must decide what is *important*.
• Downstream Alignment: This objective aligns pre-training directly with the downstream task of abstractive summarization.

Summary Generation as Pre-training is learning to condense — teaching the model to extract and synthesize the core meaning of a document.