Random Span Length

Random Span Length is a masking parameter used in span-based pre-training objectives (like SpanBERT and T5) — instead of masking spans of a fixed size, the length of each masked span is sampled from a probability distribution (typically geometric or uniform) to expose the model to missing information of varying granularity.

Distribution Details

• Geometric Distribution: Most common choice (e.g., SpanBERT uses $l \sim Geo(0.2)$) — skews toward shorter spans but allows occasional long spans.
• Mean Length: Typically targeted around 3 subword tokens — balancing single words and short phrases.
• Clamping: Spans are often clamped to a maximum length (e.g., 10) to prevent masking practically the entire sequence.
• Diversity: Ensures the model learns to handle both local (short span) and global (long span) context reconstruction.

Why It Matters

• Robustness: Evaluating on variable-length missing information makes the representation more robust.
• Realism: Real-world noise or missing data isn't fixed-length — random lengths simulate diverse corruption.
• Generalization: Prevents the model from overfitting to a specific "missing hole size" heuristic.

Random Span Length is variable-sized holes — sampling mask lengths from a distribution to train models on diverse reconstruction challenges.