Cache Invalidation
Cache invalidation is the process of removing or updating stale entries from a cache when the underlying data changes. It is famously considered one of the two hard problems in computer science (along with naming things and off-by-one errors) because getting it wrong leads to serving outdated, incorrect data.
Why Cache Invalidation is Challenging
- Consistency vs. Performance: Aggressive invalidation keeps data fresh but reduces cache hit rates. Conservative invalidation improves performance but risks stale data.
- Distributed Caches: In distributed systems, ensuring all cache nodes invalidate consistently and simultaneously is difficult.
- Hidden Dependencies: Data changes may ripple through multiple cached entries in non-obvious ways.
Invalidation Strategies
- Time-Based (TTL): Set a Time to Live on each cache entry — it's automatically removed after expiration. Simple and effective for data that can tolerate some staleness. TTL values: seconds for real-time data, hours for relatively stable data, days for static content.
- Event-Based: Invalidate cache entries when the source data changes. Requires an event system (pub/sub, webhooks, database triggers) to notify the cache.
- Write-Through: When data is updated, the cache is updated simultaneously — no stale entries, but adds write latency.
- Manual Invalidation: Explicitly clear or update specific cache entries when you know the data has changed.
- Version-Based: Include a version number in cache keys. When data changes, increment the version — old cache entries naturally become unreferenced.
AI-Specific Considerations
- Model Updates: When a model is updated, all cached responses should be invalidated because the new model may produce different answers.
- RAG Source Updates: When retrieval documents are updated, cached RAG results need invalidation.
- Semantic Cache: Invalidating entries in a semantic cache requires understanding which cached responses are affected by a data change.
- System Prompt Changes: Modifying system prompts should invalidate all response caches.
Best Practice: Use TTL as a safety net (entries eventually expire even if event-based invalidation fails) combined with event-based invalidation for time-sensitive data changes.