Sign in Get started

Database Caching at Scale · Capstone Case Studies

Capstone: Postmortem — Stale Prices After a Cache Failover

Reconstruct an incident from timelines and code, find the root causes and write the remediation plan.

Scenario

Example company: an electronics retailer. On Tuesday, 18:42, the Redis primary holding product prices crashed; Sentinel promoted a replica in 9 seconds. From 18:43 until 00:40, 1,370 products showed old prices; 212 orders were placed at outdated (lower) prices, costing about $41,000. The database always had the right prices.

Evidence

  • Price keys: price:{sku}, SET … EX 21600 (6 hours), filled on miss from a read replica.
  • Price updates: the pricing service updates Postgres, then calls DEL price:{sku} inside the database transaction block (before commit).
  • 18:30–18:42: a repricing job changed 4,000 prices in a burst.
  • Redis replication lag during the burst peaked at ~3 seconds; Postgres replica lag peaked at 4 seconds.
  • Checkout uses the cached price for the order total.
def reprice(sku, price):
    with db.transaction():
        db.execute("UPDATE products SET price = %s WHERE sku = %s", (price, sku))
        r.delete(f"price:{sku}")                          # inside the transaction
        audit.log(sku, price)

def get_price(sku):
    v = r.get(f"price:{sku}")
    if v is None:
        v = replica.query_one("SELECT price FROM products WHERE sku = %s", (sku,))
        r.set(f"price:{sku}", v, ex=21600)
    return v

Deliverables

  1. List every mechanism that could have produced stale prices here, and estimate which ones mattered.
  2. Explain why the staleness lasted until 00:40.
  3. Write the remediation plan: immediate, short-term and long-term.
  4. Define the monitoring and tests that would catch this class of bug.

Hints

  • There are at least three independent ways this code caches old values.
  • What decides the price a customer pays?

Draft each deliverable before revealing the model answer and rubric.

Further reading: Scaling Memcache at Facebook · Redis replication

Deliverables

Which mechanisms produced stale prices?

Model answer

  1. Lost deletes on failover: Redis replication is async; deletes acknowledged by the old primary during the repricing burst (up to ~3 s of lag) never reached the promoted replica → old values resurrected.
  2. Delete before commit: readers missing between the DEL and the commit re-cached the old price from the database.
  3. Fill from a lagging replica: misses right after commits read the Postgres replica (up to 4 s behind) and cached the old price.
  4. Stale-set race in general cache-aside (slow reader sets after a delete). During the burst, (2) and (3) re-cached many old prices; the failover (1) resurrected the rest. All were then locked in by the 6-hour TTL.

Rubric

Full marks: all of async replication/lost deletes, delete-before-commit, replica fills (and ideally the general stale-set race), with reasoning about the burst. Partial: only the failover.

Why did it last until 00:40?

Model answer

The 6-hour TTL: entries cached with old prices around 18:40 lived until about 00:40. Nothing else (no staleness probes, no post-failover invalidation) corrected them earlier.

Rubric

Look for: linking the duration to the TTL and the absence of detection/repair. Misses: blaming Sentinel's failover time.

Remediation plan.

Model answer

  • Immediate: flush/bump the price generation; honour or refund mispriced orders per policy.
  • Short term: delete after commit (on-commit hook); fill misses from the primary; TTL for prices ≤ 5 minutes with jitter; checkout reads the authoritative price from the database in the order transaction.
  • Long term: versioned price entries with set-if-newer (Lua); invalidation via outbox/CDC (covers every writer, replayable after failovers); on every Redis failover automatically bump generations and replay the last N minutes of invalidations; staleness probes.

Rubric

Look for: fixing all mechanisms, making checkout authoritative, and automated post-failover recovery. Misses: only shortening the TTL.

Monitoring and tests.

Model answer

  • Continuous staleness probes for prices (sample SKUs; compare cache vs primary; alert if mismatched > 30 s).
  • Alerts on Redis failover events tied to an automatic generation bump; dashboard of invalidation pipeline lag.
  • Tests: race tests for delete-before-commit and stale sets with real Redis; a chaos test that fails over Redis during a repricing burst and asserts no stale prices after N seconds.
  • Business alarm: order price ≠ current database price at checkout.

Rubric

Look for: probes, failover-aware automation, race and chaos tests, and a business-level check. Misses: only infrastructure metrics.

Open in the app to mark lessons complete, take notes and track your progress.