Discourse Marker Prediction

Discourse Marker Prediction is a self-supervised pre-training task where the model must predict missing or masked discourse markers (e.g., "however", "therefore", "because") — forcing the model to understand the logical relationship between text segments (contrast, causality, elaboration) rather than just keyword co-occurrence.

Mechanism

• Selection: Identify discourse markers in the text using a predefined list.
• Masking: Hide these markers.
• Task: The model predicts the correct marker from a set of candidates.
• Example: "He studied hard [MASK] he failed the test." -> Predict "yet" or "but".

Why It Matters

• Coherence: Discourse markers are the glue of logic — predicting them requires understanding the *argument* structure.
• NLI: Improves performance on Natural Language Inference tasks (Entailment/Contradiction).
• Discrimination: Helps models distinguish between correlation ("and") and causation ("because").

Discourse Marker Prediction is logic gap-filling — training the model to understand the logical flow of text by predicting the connecting words that define relationships.