Max Length

Max length is the hard upper bound on generated token count for a response or completion request - it is the primary guardrail against unbounded decoding.

What Is Max length?

• Definition: Configured maximum number of new tokens allowed per generation call.
• Boundary Role: Acts as final safety cap even when other stop conditions fail.
• Interaction: Works alongside EOS detection, stop sequences, and timeout policies.
• Deployment Context: Commonly set per endpoint, model tier, or customer plan.

Why Max length Matters

• Cost Control: Caps token usage for predictable billing and infrastructure load.
• Latency Limits: Prevents excessively long responses that violate user expectations.
• Abuse Resistance: Reduces impact of prompts designed to force runaway generation.
• Capacity Planning: Simplifies throughput forecasting and queue management.
• UX Consistency: Keeps responses within expected length ranges per product surface.

How It Is Used in Practice

• Tiered Limits: Set different max lengths for chat, analysis, and background jobs.
• Prompt Alignment: Pair limits with instructions to produce concise or detailed outputs.
• Monitoring: Track truncation rates to detect limits that are too restrictive.

Max length is a mandatory control for safe and economical inference - proper max-length policy balances completeness, cost, and latency.