Cache Failover and Lost Invalidations

Scenario

Example app: a store keeps product prices in Redis (primary + replica, Sentinel for failover). Price changes commit to Postgres, then DEL price:{id} runs. One evening the Redis primary's host dies.

What you see in production

After Sentinel promotes the replica, a few hundred products show prices from before the latest change — for the full 6-hour TTL. The database is correct; the deletes had been acknowledged by the old primary.

The naive code

def change_price(pid, price):
    db.update_price(pid, price)
    r.delete(f"price:{pid}")            # acknowledged by the primary; replication to the replica is async
# price keys use ex=6 * 3600

Reproduce it

A primary acknowledges deletes before the replica has them; the primary dies; the replica is promoted:

primary = {"price:1": 100, "price:2": 50, "price:3": 70}
replica = dict(primary)
in_flight = []

def delete(key):
    primary.pop(key, None)
    in_flight.append(key)              # replicated later (async)

def replicate(n):
    for _ in range(min(n, len(in_flight))):
        replica.pop(in_flight.pop(0), None)

delete("price:1"); delete("price:2"); delete("price:3")   # three price changes
replicate(1)                                                # only the first delete reached the replica
print("primary dies; promoting replica")
print("cache now serves:", replica, "<- price:2 and price:3 are stale again")

primary dies; promoting replica
cache now serves: {'price:2': 50, 'price:3': 70} <- price:2 and price:3 are stale again

Root cause

Redis replication is asynchronous: the primary acknowledges writes (including DEL) before replicas receive them. A failover can therefore lose recent writes. For a cache, a lost DEL is worse than a lost SET: it resurrects a value that was deliberately removed, and a long TTL keeps it alive. Redis's WAIT command can block until replicas acknowledge, but it doesn't make Redis strongly consistent.

The fix

• Short TTLs for values that must not be stale (prices: minutes, not hours).
• Versioned values (store the row version): readers compare with a cheap version source, or writers bump a generation that makes old entries unreachable.
• On failover, treat the cache as suspect: bump a global or per-keyspace generation (instant logical flush) or flush the affected keyspace, then let it warm up with stampede protection.
• Replay recent invalidations from the change stream (CDC/outbox) after a failover — the pipeline from event-driven invalidation makes this possible.

def on_redis_failover():
    r.incr("gen:prices")                          # every price key embeds the generation → all old entries unreachable
    replay_invalidations(since=now() - timedelta(minutes=5))   # re-issue recent deletes from the outbox/CDC log

# make important deletes wait for at least one replica (best effort, adds latency)
DEL price:42
WAIT 1 100        # wait up to 100 ms for 1 replica to acknowledge

// Lettuce: subscribe to Sentinel +switch-master events and bump the cache generation when they occur
sentinelConnection.reactive().observeChannels().filter(m -> m.getChannel().equals("+switch-master"))
    .subscribe(m -> cacheGeneration.increment());

Trade-offs

• Generation bumps after failover cause a burst of misses — combine with stampede protection and bounded database fallback.
• WAIT adds write latency and still isn't a consistency guarantee (the replica that acknowledged may not be the one promoted).
• Very short TTLs reduce staleness but lower hit ratio; use them only for values where staleness is costly.

Detect it in production

• Alert on Redis failovers (Sentinel/Cluster events) and correlate with staleness probes.
• Staleness probes for critical keys (compare with the database) run continuously, not only after incidents.

Key points

• Redis replication is async; failovers can lose recent writes, including deletes.
• Lost deletes resurrect stale values for the full TTL.
• Keep critical TTLs short, version values, and invalidate or bump generations after failover.