Why is deleting the cache key before the transaction commits a bug?

Until commit, other readers still see the old row. A reader that misses in that window loads the old value and re-caches it, undoing the invalidation.

How can a 1-second replica lag cause 10 minutes of stale cache?

A cache miss right after the write is filled from the replica, which hasn't applied the write yet; the old value is then cached for the full TTL.

What is a delayed second delete and when does it help?

Deleting the key again after the maximum expected replica lag, to remove values that were re-cached from stale sources in between. It narrows the window; it isn't a strict guarantee.

Caching with Transactions and Replicas — Database Caching at Scale

Scenario

Example app: user profiles are cached in Redis. Reads come from a read replica; writes go to the primary inside a transaction that also updates search and audit tables.

What you see in production

After users update their display name, a few percent of them see the old name for 10 minutes (the TTL). It's worse during busy periods, when replica lag reaches a second or two. The code deletes the cache key on every update.

The naive code

@transaction.atomic
def rename(user_id, name):
    r.delete(f"profile:{user_id}")                       # (1) invalidated BEFORE commit
    Profile.objects.filter(user_id=user_id).update(name=name)
    write_audit_log(user_id, name)                       # transaction still open…

def get_profile(user_id):
    if (hit := r.get(f"profile:{user_id}")) is not None:
        return json.loads(hit)
    p = Profile.objects.using("replica").get(user_id=user_id)   # (2) filled from a lagging replica
    r.set(f"profile:{user_id}", to_json(p), ex=600)
    return p

@Transactional
@CacheEvict(value = "profiles", key = "#userId", beforeInvocation = true)   // evicts before the update commits
public void rename(long userId, String name) { ... }

Reproduce it

Two timelines: invalidate-before-commit, and fill-from-replica-after-commit:

Two ways to re-cache the old valueTry it yourself

def run(label, steps):
    primary, replica, cache = {"name": "Ada"}, {"name": "Ada"}, {}
    for step in steps:
        step(primary, replica, cache)
    print(f"{label:40} cache={cache.get('name')!r} primary={primary['name']!r}")

delete = lambda p, r, c: c.pop("name", None)
reader_from_primary = lambda p, r, c: c.setdefault("name", p["name"])   # miss → SELECT → SET
reader_from_replica = lambda p, r, c: c.setdefault("name", r["name"])
commit = lambda p, r, c: p.update(name="Ada L.")
replicate = lambda p, r, c: r.update(name=p["name"])

run("delete before commit", [delete, reader_from_primary, commit])
run("delete after commit, fill from replica", [commit, delete, reader_from_replica, replicate])
run("delete after commit, fill from primary", [commit, delete, reader_from_primary])

Output

delete before commit                     cache='Ada' primary='Ada L.'
delete after commit, fill from replica   cache='Ada' primary='Ada L.'
delete after commit, fill from primary   cache='Ada L.' primary='Ada L.'

Root cause

Invalidating before commit: until the transaction commits, other readers still see the old row (isolation). A reader that misses in that window re-caches the old value.
Filling from a replica: right after the commit, the replica may not have the change yet. A miss filled from it caches the old value for the whole TTL — the replica's lag is converted into much longer cache staleness.

The fix

Invalidate after commit (on-commit hooks, transactional events, transaction-aware cache managers).
Fill the cache from the primary — or from a replica only once it has replayed past the write (see Read-Your-Writes).
Optionally delete twice: once after commit and again after max replica lag, to catch fills that slipped through.

def rename(user_id, name):
    with transaction.atomic():
        Profile.objects.filter(user_id=user_id).update(name=name)
        write_audit_log(user_id, name)
        transaction.on_commit(lambda: invalidate_profile(user_id))

def invalidate_profile(user_id):
    r.delete(f"profile:{user_id}")
    delete_again_later.apply_async((f"profile:{user_id}",), countdown=2)   # > max replica lag

def get_profile(user_id):
    if (hit := r.get(f"profile:{user_id}")) is not None:
        return json.loads(hit)
    p = Profile.objects.using("default").get(user_id=user_id)   # fill from the primary
    r.set(f"profile:{user_id}", to_json(p), ex=600)
    return p

@Transactional
public void rename(long userId, String name) {
    profiles.rename(userId, name);
    events.publishEvent(new ProfileChanged(userId));
}

@TransactionalEventListener(phase = TransactionPhase.AFTER_COMMIT)
public void evict(ProfileChanged e) { cache.evict(e.userId()); }

DB::transaction(function () use ($userId, $name) {
    Profile::where('user_id', $userId)->update(['name' => $name]);
    DB::afterCommit(fn () => Cache::forget("profile:$userId"));
});

await prisma.$transaction(async (tx) => { await tx.profile.update({ where: { userId }, data: { name } }); });
await redis.del(`profile:${userId}`);                        // after the transaction resolved
setTimeout(() => redis.del(`profile:${userId}`), 2000);     // second delete after max replica lag

Trade-offs

Filling from the primary moves miss traffic to the primary; acceptable when the hit ratio is high, risky during a cold start.
Delayed second deletes need a reliable scheduler (a queue), or they're lost on restarts.
After-commit hooks don't run if the process dies between commit and hook — keep TTLs as the backstop, or invalidate from the outbox/CDC (later lesson).

Detect it in production

Staleness probes right after writes (write → read through the normal path → compare).
Correlate stale reports with replica lag spikes.

Key points

Invalidate after commit, never before.
Don't fill caches from lagging replicas after a write; fill from the primary or wait for the replica.
Keep TTLs (and optionally a delayed second delete) as backstops.