Sentence Scrambling

Sentence Scrambling is a pre-training objective where the sentences of a document are randomly reordered, and the model must identify the correct order or predict the position of a specific sentence — similar to Sentence Permutation but often formulated as a classification or ranking task rather than generation.

Variants

• Reordering: Generatively reconstruct the document (BART).
• Binary Classification: "Do these two sentences appear in this order?" (ALBERT SOP).
• Ranking: "Which of these 5 candidates is the correct next sentence?"
• Position Prediction: "What is the absolute position of this sentence in the document?"

Why It Matters

• Structure Learning: Forces learning of narrative structure (Introduction → Body → Conclusion).
• Long-Range Dependencies: To order sentences correctly, the model must track entities and themes across the whole document.
• Coherence: Essential for tasks like summarization and story generation where flow matters.

Sentence Scrambling is putting the story back together — teaching the model document-level coherence by forcing it to reassemble jumbled sentences.