Next Sentence Prediction

Next Sentence Prediction (NSP) is a pre-training objective introduced in BERT where the model predicts whether a given sentence B immediately follows sentence A in the original text — a binary classification task designed to teach the model relationships between sentences (discourse, entailment, continuity).

NSP Details

• Input: Pairs of sentences (A, B) packed together: `[CLS] A [SEP] B [SEP]`.
• Positive Sample (IsNext): B is the actual next sentence from the corpus (50% probability).
• Negative Sample (NotNext): B is a random sentence from the corpus (50% probability).
• Prediction: The `[CLS]` token embedding is fed to a classifier to output IsNext/NotNext.
• Critique: Later research (RoBERTa) showed NSP was not very effective — mostly learning topic matching rather than coherence.

Why It Matters

• Original BERT: A core component of the original BERT training recipe.
• Discourse: Intended to help with tasks like QA and NLI (Natural Language Inference) that require reasoning across sentences.
• Legacy: Largely replaced by more effective objectives (like SOP) or removed entirely in modern LLMs.

NSP is original BERT's coherence check — a binary task checking if two sentences belong together, now considered largely obsolete by improved methods.