Document Rotation

Document Rotation is a pre-training objective where a token is chosen uniformly at random, and the document is rotated so that token becomes the start — used in models like BART, this trains the model to identify the true start of a document and reconstruct the original sequence.

Mechanism

• Select Pivot: Choose a random token $t_k$ in the sequence.
• Rotate: Move tokens before $t_k$ to the end: $[t_k, dots, t_n, t_1, dots, t_{k-1}]$.
• Objective: The model (seq2seq) must accept the rotated sequence and generate the original un-rotated sequence.
• Inference: The model learns to be invariant to the starting point or to identify the logical beginning.

Why It Matters

• Start Identification: Forces identification of the introductory sentence or logical opening.
• Context Cycle: Ensures the model can handle context that wraps around — useful for sliding window approaches?
• Global Structure: Like shuffling, it forces an understanding of the document's overall structure.

Document Rotation is finding the beginning — a structural pre-training task where the model learns to identify the start of a document from a rotated version.