Observing and Testing Caches

Scenario

Example company: caches were added over years by different teams. The dashboard shows one number: overall Redis hit ratio, 92%. Incidents keep happening anyway: a key prefix whose hit ratio quietly dropped to 3% after a deploy, a stale-price bug that went unnoticed for weeks.

What you see in production

Postmortems repeatedly find that the cache problem was visible in data nobody looked at. Unit tests mock the cache entirely, so races and invalidation bugs are never tested.

The naive code

def test_get_product(mocker):
    mocker.patch("app.cache.get", return_value=None)    # cache behaviour never exercised
    assert get_product(1).id == 1

Reproduce it

An overall hit ratio can hide a broken keyspace. Compute per-prefix ratios from access counters:

stats = {          # prefix: (hits, misses) over the last hour
    "product": (9_200_000, 300_000),
    "session": (4_100_000, 20_000),
    "category": (35_000, 1_100_000),      # a deploy changed the key format: nothing hits anymore
}
hits = sum(h for h, _ in stats.values())
total = sum(h + m for h, m in stats.values())
print(f"overall hit ratio: {hits / total:.1%}")
for prefix, (h, m) in stats.items():
    ratio = h / (h + m)
    print(f"  {prefix:9} {ratio:6.1%}  {'<- ALERT' if ratio < 0.5 else ''}")

overall hit ratio: 90.4%
  product    96.8%  
  session    99.5%  
  category    3.1%  <- ALERT

Root cause

Aggregate metrics average away the failures that matter, and caches are usually tested only as "works when mocked". Cache correctness problems are concurrency and time problems — they need instrumentation per keyspace and tests that control interleavings and clocks.

The fix

Instrument per keyspace (prefix): hits, misses, latency, payload size, errors, evictions, and staleness probes for critical data.

def cached(prefix, key, loader, ttl):
    start = time.perf_counter()
    raw = r.get(f"{prefix}:{key}")
    metrics.timing("cache.get.ms", (time.perf_counter() - start) * 1000, tags={"prefix": prefix})
    if raw is not None:
        metrics.increment("cache.hit", tags={"prefix": prefix})
        return json.loads(raw)
    metrics.increment("cache.miss", tags={"prefix": prefix})
    value = loader()
    r.set(f"{prefix}:{key}", json.dumps(value), ex=ttl)
    return value

def test_no_stale_set_after_update(redis_container, db):
    # reader misses and pauses after reading the old value
    reader = ControlledReader(pause_after_db_read=True)
    reader.start(product_id=1)
    update_price(1, 80)                                  # commit + delete
    reader.resume()                                      # tries to set the old value
    assert get_cached_price(1) in (None, 80)             # versioned set must reject 100

@Container static GenericContainer<?> redis = new GenericContainer<>("redis:7").withExposedPorts(6379);

@Test void entryExpiresAfterTtl() {
    FakeTicker ticker = new FakeTicker();
    Cache<Long, String> l1 = Caffeine.newBuilder().expireAfterWrite(Duration.ofSeconds(2)).ticker(ticker::read).build();
    l1.put(1L, "a");
    ticker.advance(Duration.ofSeconds(3));
    assertThat(l1.getIfPresent(1L)).isNull();
}

Dashboards to have: hit ratio per prefix, Redis latency and ops per shard, memory and evictions, database QPS attributed to cache misses, invalidation pipeline lag, staleness probe failures.

Trade-offs

• Per-key metrics explode cardinality — aggregate by prefix and sample individual hot keys.
• Real-Redis tests (Testcontainers) are slower than mocks but catch serialization, TTL and script bugs; keep a small, focused set.
• Staleness probes cost a database read each — sample.

Detect it in production

• Alerts on per-prefix hit-ratio drops after deploys, on eviction spikes, and on staleness probe failures.
• A cache section in every postmortem template: which metrics would have shown it earlier?

Key points

• Measure hit ratio, latency, size and errors per keyspace — not just overall.
• Probe staleness for critical data continuously.
• Test caches with real Redis, controlled interleavings and fake clocks.