Cache Hit Rate
Cache hit rate is the percentage of requests that are successfully served from the cache (hits) versus the total number of requests (hits + misses). It is the primary metric for evaluating cache effectiveness.
Formula
$$\text{Hit Rate} = \frac{\text{Cache Hits}}{\text{Cache Hits} + \text{Cache Misses}} \times 100\%$$
Interpreting Hit Rate
- >90%: Excellent — the cache is highly effective. The vast majority of requests are served from cache.
- 70–90%: Good — the cache is working well but there may be opportunities to improve.
- 50–70%: Moderate — consider if the cache strategy matches the access patterns.
- <50%: Poor — the cache may be too small, eviction policy may be wrong, or the workload may not benefit from caching.
Factors That Affect Hit Rate
- Cache Size: Larger caches store more entries and have higher hit rates, but cost more memory.
- Eviction Policy: LRU (Least Recently Used), LFU (Least Frequently Used), and other policies determine which entries to remove when the cache is full.
- TTL (Time to Live): Shorter TTLs cause entries to expire before they can be reused; longer TTLs risk serving stale data.
- Access Pattern: Workloads with high temporal locality (recently accessed items are likely to be accessed again) benefit most from caching.
- Cache Key Design: Using too-specific keys (exact prompt match) reduces hit rates vs. semantic matching.
Cache Hit Rate for LLM Applications
- Exact Match Caching: Typically 5–15% hit rate for conversational AI (queries vary widely).
- Semantic Caching: Can achieve 20–40% hit rate by matching semantically similar queries.
- FAQ/Support Bots: Often 50–80% hit rate because users ask the same questions repeatedly.
- KV Cache: ~100% hit rate within a single generation (each new token reuses all previous KV entries).
Monitoring
- Track hit rate over time — sudden drops may indicate cache invalidation issues, workload changes, or deployment problems.
- Monitor by cache tier (L1/L2) and by query type to identify optimization opportunities.
Cache hit rate directly translates to cost savings and latency reduction — even a 10% improvement can significantly reduce LLM API spending.