102 technical terms and definitions
kv cache, key-value cache, llm inference cache, paged attention, gqa, mqa, kv quantization
Key-value cache stores attention key and value tensors already computed for earlier tokens during autoregressive inference. It prevents every decode step from recomputing all…
kv cache quantization, kv cache eviction, attention memory, gqa, mqa, streamingllm, h2o, low rank kv cache
KV cache compression reduces the memory footprint or bandwidth of stored attention keys and values used during autoregressive inference. Long contexts and many concurrent requests…