Sentence Permutation

Sentence Permutation is a pre-training objective where the order of sentences in a document is randomly shuffled, and the model must reconstruct the original order — used in models like BART and PEGASUS to teach the model about document-level structure, coherence, and flow logic.

Permutation Mechanism

• Shuffling: Break document into sentences $S_1, S_2, dots, S_n$. Randomly permute them to $S_{p1}, S_{p2}, dots, S_{pn}$.
• Reconstruction: The model (typically seq2seq) treats the shuffled text as input and must generate the sentences in the correct original order.
• Difficulty: Extremely challenging for long documents — requires understanding logical progression, anaphora, and narrative arc.
• BART: Uses sentence permutation as one of detailed denoising objectives.

Why It Matters

• Coherence: Forces the model to understand *why* sentence A follows sentence B — logic and causality.
• Summarization: Excellent pre-training for summarization — requires understanding global document structure.
• Long Context: Encourages attention to long-range dependencies across the entire input.

Sentence Permutation is unscrambling the story — a document-level objective that forces the model to learn structure and coherence by reordering shuffled sentences.