Length Penalty

Length penalty adjusts sequence scores in beam search to control output length preferences. Problem: Log probabilities accumulate negatively - longer sequences have lower scores, biasing toward short outputs. Solution: Normalize by length: score = log_prob / length^α. Alpha values: α = 0 (no normalization, favor short), α = 1 (linear normalization), α > 1 (favor longer sequences), α < 1 (mild length compensation). Google's formula: lp(Y) = ((5 + |Y|)/(5 + 1))^α - smoothed length penalty avoiding division by zero for short sequences. Implementation: Apply during beam selection and final ranking. Use cases: Translation (α ≈ 0.6-0.8 for balanced length), summarization (adjust based on desired length), structured outputs. Related controls: Max/min length constraints, length-conditional training and sampling. Alternatives: Direct length tokens in prompt, length-conditioned decoding, explicit length prediction. Best practices: Tune α empirically on validation set, different tasks need different settings, combine with other quality metrics for final selection.