What Is Attention Mechanism
The attention mechanism is the calculation that lets a transformer decide, for every token it's producing, which earlier tokens actually matter right now — and it's exactly what the KV cache from the earlier inference discussion is built to store. Before transformers, sequence models mostly processed tokens strictly in order, carrying forward a single compressed summary of everything seen so far. Attention replaces that bottleneck with something more direct: every token gets to look back at every other token and weigh how relevant each one is to the current step. That weighing is what makes transformers so good at long-range context — and it's also precisely why serving them is so memory-hungry, since "looking back at every other token" means storing something for every one of those tokens somewhere.
Every token produces three vectors — a query, a key, and a value — and attention is fundamentally just comparing queries against keys to decide how much of each value to use. The query represents what the current token is "looking for." The key represents what each earlier token "offers" as a matching signature. The model compares the current query against every earlier key, turns those comparisons into a set of weights (via a softmax, so they sum to a probability distribution), and then blends the earlier tokens' value vectors together using those weights. A token attending strongly to an earlier one means its key matched well against the current query — the model decided that earlier token is highly relevant right now.
This is exactly why the KV cache exists and exactly why it keeps growing: the "K" and "V" in KV cache are literally the keys and values every earlier token produced. Since a new token's query has to be compared against every earlier token's key to attend correctly, those keys (and the values they pair with) have to stay available in memory for as long as the sequence continues. Recomputing them from scratch for every new token would mean redoing the full forward pass over the entire history each time — recognized as wasteful in the earlier inference discussion — so instead they're computed once, cached, and simply reused and extended. That's the whole reason a long conversation or long document makes decode heavier: the set of keys and values a new token's query has to sweep across just keeps getting bigger.
| Component | What It Represents | Computed | Stored in KV Cache? |
|---|---|---|---|
| Query (Q) | What the current token is looking for | Fresh, every new token | No — used once, then discarded |
| Key (K) | What each token offers as a match signature | Once, when that token is first processed | Yes — reused by every later token's query |
| Value (V) | The actual content blended into the output | Once, when that token is first processed | Yes — reused by every later token's query |
Understanding attention this way makes the whole memory-bandwidth story from the last two entries click into place at the hardware level. Every decode step has to pull every stored key and value back out of memory to compare against the current query — full-precision, that's a lot of bytes moving for every single token generated, which is exactly the load quantization was shown to shrink. Techniques like multi-query and grouped-query attention (sharing keys and values across multiple attention heads instead of storing separate copies for each) exist specifically to reduce how much of this cache has to be read per step, which is why they show up constantly in modern AI accelerator and model-architecture discussions: attention's elegance is also, unavoidably, its memory bill.