Beam Search Decoding

When a language model finishes a forward pass it does not hand you a word. It hands you a probability distribution over its entire vocabulary, and *decoding* is the policy you use to turn that distribution into the next token. The model is the same every time; the sampler is the dial you actually control at inference. Two people running the identical model can get a crisp deterministic answer or a wild creative riff purely by choosing different decoding settings.

Greedy decoding takes the single most likely token at every step. It is fast, reproducible, and locally optimal, but it is also myopic: always grabbing the top token can walk the model into bland, repetitive, or degenerate loops because the globally best sentence sometimes starts with a locally second-best word.

Beam search widens the search by keeping the *k* most probable partial sequences alive at once, extending all of them and pruning back to the top *k* each step. It reliably finds higher-probability full sequences and is the workhorse of machine translation and summarization, where there is roughly one correct answer. For open-ended generation it tends to produce safe, generic text and can collapse the beams onto near-duplicates.

Temperature reshapes the distribution before you sample from it by dividing the logits by a scalar T inside the softmax. T below 1 sharpens the distribution and concentrates mass on the top tokens (more conservative); T above 1 flattens it and hands probability to the long tail (more diverse and more error-prone). T = 1 leaves the model's native distribution untouched, and T approaching 0 collapses back to greedy.

Top-k sampling truncates the candidate set to the k highest-probability tokens, renormalizes, and samples from just those. It kills the long tail of absurd tokens, but a fixed k is a blunt instrument: when the model is confident, k is too generous, and when it is unsure, k is too stingy.

Top-p (nucleus) sampling truncates by cumulative probability mass instead of by count — it keeps the smallest set of tokens whose probabilities sum to p (say 0.9) and samples from that. The candidate set breathes: it shrinks to a couple of tokens when the model is certain and expands to dozens when it is not, which is why top-p is the most widely used default for chat and creative generation. In practice teams stack a modest temperature with top-p and leave the rest alone.

MethodDeterminismDiversityBest forFailure mode
GreedyDeterministicNoneShort factual answers, codeRepetition, blandness
Beam search (k)DeterministicLowTranslation, summarizationGeneric, near-duplicate beams
Temperature (T)StochasticTunableGlobal creativity knobHigh T -> incoherence
Top-kStochasticMediumCutting the absurd tailFixed k mis-sizes the set
Top-p / nucleusStochasticAdaptiveChat, open-ended textVery high p -> drift
Sampling — Turning Next-Token Probabilities into Text the model scores every token; the decoding strategy decides which one to actually emit — and how much risk to take Top-k (k = 3) keep a fixed number of candidates, renormalize, sample kept tail discarded Top-p / nucleus (p = 0.90) smallest set whose probs sum to p — count adapts to confidence the nucleus Temperature: softmax(z / T) T < 1 sharpens T = 1 raw T > 1 flattens divide logits by T before softmax: low = safe & sharp, high = diverse Greedy & beam (deterministic) Greedy takes the single most likely token every step — fast, but bland and repetitive. Beam keeps the top-B partial sequences and scores whole- sentence likelihood: good for translation, dull for open-ended generation. Temperature: the risk dial Divides the logits by T before the softmax. T→0 approaches greedy (sharp, safe); T = 1 is the model's raw distribution; T > 1 flattens it, raising surprise and diversity at the cost of coherence. The one knob most people actually tune. Top-k vs Top-p (truncation) Both chop off the unreliable tail before sampling. Top-k keeps a fixed count; top-p keeps a variable one — the smallest set covering probability p — so it widens when the model is unsure, narrows when confident. Nucleus + temperature is the default.

The mistake most people make is treating decoding as an afterthought — a single "temperature" slider to nudge when output feels off. It is better understood as the interface between a fixed probabilistic model and the text you actually want. Greedy and beam search ask *what is most probable*; temperature, top-k, and top-p ask *how much of the model's uncertainty should I let through, and in what shape*. Read decoding through a shape-the-distribution lens rather than a pick-the-best-word lens, and every parameter stops being a magic number and becomes a deliberate statement about how much risk you want the model to take on each token.

Take beam search decoding further

Ask the copilot about this term, or have our engineers assess it against your process.

Ask CFSGPT about this → Request a technical assessment → Open the CFS app