Constrained Decoding

Constrained decoding forces LLM outputs to follow specific rules, formats, or grammars. Mechanism: During each token selection, mask invalid tokens based on constraints, only allow valid continuations, constraints can be regular expressions, context-free grammars, or schema-based. Use cases: Guaranteed JSON output, SQL generation, code in specific syntax, formatted responses, controlled vocabulary. Implementation approaches: Grammar-based (define valid token sequences), regex-guided (match pattern during generation), schema-constrained (JSON Schema, Pydantic models), finite state machines. Tools: Outlines (grammar-constrained generation), Guidance (structured prompting), llama.cpp grammars, NVIDIA TensorRT-LLM constraints. Performance: Adds overhead for constraint checking, but prevents retry loops from format failures. JSON generation: Define JSON grammar, only allow valid JSON tokens at each step, guarantees parseable output. Trade-offs: Constraints may force unnatural completions, effectiveness depends on model's alignment with constraints. Essential for production systems requiring structured, parseable outputs.