Cache Avalanches and Cold Starts

Scenario

Example app: a news site serves 30,000 requests per second with a 99% cache hit ratio; the database handles the 300 misses per second comfortably. One night a Redis node restarts (no persistence) during peak traffic.

What you see in production

With an empty cache, the database suddenly receives 30,000 queries per second — 100× its normal load. It saturates, requests time out, and because timed-out requests never fill the cache, the cache never warms up: the site stays down until traffic is throttled manually.

The naive code

The cache is assumed to always be there:

def article(aid):
    try:
        hit = r.get(f"article:{aid}")
    except redis.ConnectionError:
        hit = None                         # fall back to the database for EVERY request
    if hit:
        return json.loads(hit)
    data = db.fetch_article(aid)           # unbounded
    r.set(f"article:{aid}", json.dumps(data), ex=3600)
    return data

Reproduce it

Database load with a warm cache, a cold cache, and a cold cache protected by a concurrency limit on database fallbacks plus warming of hot keys:

RPS, DB_CAPACITY = 30_000, 3_000            # database can serve 3,000 queries/s

def unprotected(hit_ratio):
    return RPS * (1 - hit_ratio)

print(f"warm (99% hits):  {unprotected(0.99):>8,.0f} db qps")
print(f"cold (0% hits):   {unprotected(0.00):>8,.0f} db qps  -> {unprotected(0) / DB_CAPACITY:.0f}× capacity")

hit_ratio, second = 0.0, 0
while hit_ratio < 0.95:
    admitted = min(RPS * (1 - hit_ratio), DB_CAPACITY * 0.8)   # bulkhead: cap fallback queries
    shed = RPS * (1 - hit_ratio) - admitted                     # served stale/degraded or 503
    hit_ratio = min(0.99, hit_ratio + admitted / RPS * 0.6)     # admitted misses fill the cache
    second += 1
    if second % 4 == 0 or hit_ratio >= 0.95:
        print(f"t={second:2}s protected: db qps {admitted:,.0f}, degraded {shed:,.0f}, hit ratio now {hit_ratio:.0%}")

warm (99% hits):       300 db qps
cold (0% hits):     30,000 db qps  -> 10× capacity
t= 4s protected: db qps 2,400, degraded 23,280, hit ratio now 19%
t= 8s protected: db qps 2,400, degraded 17,520, hit ratio now 38%
t=12s protected: db qps 2,400, degraded 11,760, hit ratio now 58%
t=16s protected: db qps 2,400, degraded 6,000, hit ratio now 77%
t=20s protected: db qps 2,400, degraded 240, hit ratio now 96%

Root cause

A high hit ratio means the database is sized for the misses, not the traffic. Losing the cache (restart, failover, flush, a deploy that changes every key) multiplies database load by 1 / (1 − hit ratio). Without limits, the overloaded database fails the very requests that would refill the cache. Facebook's memcache paper describes protecting backends from failed cache servers with a small Gutter pool (about 1% of memcached servers) that takes over for failed servers, and warming new "cold" clusters from warm ones (Scaling Memcache at Facebook).

The fix

1. Bound database fallback: a concurrency limit/bulkhead for cache-miss queries; beyond it, serve stale data, a degraded page or a fast 503 — never queue unboundedly.
2. Keep caches warm through restarts: replicas with failover, persistence (RDB/AOF) where restart warm-up matters, and rolling restarts.
3. Warm deliberately: preload hot keys before shifting traffic; avoid deploys that change every key at once (roll key-version changes gradually).
4. Jitter TTLs so a full cache never expires together.

db_fallback = threading.BoundedSemaphore(50)          # per process; size from database capacity

def article(aid):
    hit = safe_cache_get(f"article:{aid}")
    if hit:
        return json.loads(hit)
    if not db_fallback.acquire(timeout=0.05):
        return stale_or_degraded(aid)                   # e.g. last-known copy from a local disk/L1 cache
    try:
        data = db.fetch_article(aid)
    finally:
        db_fallback.release()
    safe_cache_set(f"article:{aid}", json.dumps(data), ex=jittered(3600))
    return data

Bulkhead dbFallback = Bulkhead.of("article-db-fallback",
    BulkheadConfig.custom().maxConcurrentCalls(50).maxWaitDuration(Duration.ofMillis(50)).build());
Supplier<Article> load = Bulkhead.decorateSupplier(dbFallback, () -> repo.load(id));
return Try.ofSupplier(load).recover(BulkheadFullException.class, e -> degraded(id)).get();

# keep data across restarts where warm-up time matters
appendonly yes            # AOF persistence (or RDB snapshots)
replica-read-only yes     # plus a replica + Sentinel/Cluster for failover

Trade-offs

• Shedding or degrading pages during a cold start is better than a full outage, but needs product decisions (what does a degraded page show?).
• Persistence makes restarts warm but adds disk I/O and slower restarts; many teams accept cold caches but protect the database instead.
• Warming everything is wasteful — warm the hot set (top N keys by recent traffic).

Detect it in production

• Hit ratio and database QPS on the same graph; alert on sudden hit-ratio drops.
• Redis restarts, failovers and FLUSH* commands (disable FLUSHALL in production via ACLs).
• Load tests that start with an empty cache.

Key points

• The database is sized for misses; losing the cache multiplies its load.
• Cap fallback queries, serve stale/degraded responses, and warm caches deliberately.
• Avoid synchronized expiry and mass key changes; test cold starts.