Event-Driven Invalidation with Outbox and CDC

Scenario

Example company: product data is written by the admin app, a nightly import job, a pricing microservice and occasional SQL fixes by support. The storefront, search API and recommendation service each cache product data.

What you see in production

Changes made by the import job and manual SQL never invalidate anything; prices from the pricing service invalidate only the storefront's cache, not the search API's. Each team adds cache.delete() calls in more places, and still misses some.

The naive code

Invalidation lives in individual code paths:

def update_price(pid, price):
    with transaction.atomic():
        Product.objects.filter(pk=pid).update(price=price)
        transaction.on_commit(lambda: r.delete(f"product:{pid}"))   # only this path, only this cache

UPDATE products SET price = 19.99 WHERE id = 4211;   -- no cache invalidation at all

Reproduce it

Three writers and two caches; compare per-path invalidation with a single invalidation pipeline that reads the database's change stream:

writers = ["admin app", "import job", "support SQL"]
caches = {"storefront": {}, "search-api": {}}

def per_path_invalidation(writer):
    if writer == "admin app":                       # only the admin code path deletes, and only its own cache
        caches["storefront"].pop("product:1", None)

def change_stream_consumer(change):                 # sees every committed change, whoever wrote it
    for cache in caches.values():
        cache.pop(f"product:{change['id']}", None)

for mode in ("per-path", "change stream"):
    stale = []
    for writer in writers:
        for cache in caches.values():
            cache["product:1"] = "old price"         # both caches hold the product before the write
        if mode == "per-path":
            per_path_invalidation(writer)
        else:
            change_stream_consumer({"id": 1, "by": writer})
        stale += [f"{name} after {writer}" for name, cache in caches.items() if "product:1" in cache]
    print(f"{mode:13} stale caches: {stale or 'none'}")

per-path      stale caches: ['search-api after admin app', 'storefront after import job', 'search-api after import job', 'storefront after support SQL', 'search-api after support SQL']
change stream stale caches: none

Root cause

When invalidation is a side effect of specific code paths, any other writer leaves caches stale, and every new cache must be wired into every writer. The source of truth for "what changed" is the database's commit log, not the application code.

Facebook's memcache paper describes this approach: they "deploy invalidation daemons (named mcsqueal) on every database", each inspecting committed SQL statements, extracting deletes and broadcasting them to the memcache deployment in every frontend cluster (Scaling Memcache at Facebook).

The fix

Publish changes once, from the database, and let each cache owner subscribe:

1. CDC (Debezium on the Postgres WAL / MySQL binlog) or the transactional outbox emits ProductChanged(id) for every commit, whoever made it.
2. Each service runs a small consumer that deletes (or bumps the generation of) its own keys.
3. Keep TTLs as the backstop for anything the pipeline misses.

for msg in consumer:                                   # topic: dbserver.public.products (Debezium)
    change = json.loads(msg.value)
    pid = change["after"]["id"] if change.get("after") else change["before"]["id"]
    r.delete(f"v3:product:{pid}")                      # idempotent: safe to process twice
    consumer.commit()

@KafkaListener(topics = "dbserver.public.products", groupId = "storefront-cache")
public void onChange(ConsumerRecord<String, String> record) {
    long id = extractId(record.value());
    redis.delete("v3:product:" + id);                    // each service owns its own keys
}

-- a trigger turns any update into an invalidation event, so even manual SQL is covered
CREATE OR REPLACE FUNCTION products_changed() RETURNS trigger AS $$
BEGIN
  INSERT INTO outbox (aggregate_id, type, payload)
  VALUES (NEW.id, 'ProductChanged', json_build_object('id', NEW.id));
  RETURN NEW;
END $$ LANGUAGE plpgsql;
CREATE TRIGGER products_changed AFTER INSERT OR UPDATE ON products FOR EACH ROW EXECUTE FUNCTION products_changed();

Trade-offs

• Invalidation becomes asynchronous: caches are stale for the pipeline's lag (typically under a second). Combine with read-your-writes for the writer if needed.
• More infrastructure (CDC, broker, consumers) and new failure modes: a stopped consumer means stale caches — monitor consumer lag.
• Deletes are idempotent, so at-least-once delivery is fine; ordering per key still matters if you update instead of delete.

Detect it in production

• Consumer lag per cache owner; alert when invalidations fall behind.
• Staleness probes after writes made through every path (admin, import, SQL).

Key points

• Invalidation tied to code paths misses other writers and other caches.
• Emit changes from the database (CDC or outbox), and let each cache owner consume them.
• Keep TTLs as a backstop and monitor pipeline lag.