Two-Tier Caching: In-Process L1 + Redis L2

Scenario

Example app: a pricing API runs on 40 pods. Every request needs the price rules for a product; Redis adds ~0.8 ms per call and the hot set fits easily in memory. The team adds a 60-second in-process cache (L1) in front of Redis (L2).

What you see in production

Latency drops — but after a price change, some pods serve the old price for up to a minute, so customers refreshing the page see the price flip between old and new depending on which pod answers.

The naive code

Cache<Long, PriceRules> l1 = Caffeine.newBuilder().expireAfterWrite(Duration.ofSeconds(60)).build();

public PriceRules rules(long productId) {
    return l1.get(productId, id -> redisOrDb(id));     // each pod keeps its own copy for 60 s
}

public void updateRules(long productId, PriceRules r) {
    repo.save(productId, r);
    redis.delete("rules:" + productId);                 // L2 invalidated; 39 other pods' L1 untouched
}

l1 = TTLCache(maxsize=50_000, ttl=60)
def rules(pid):
    if pid in l1:
        return l1[pid]
    l1[pid] = redis_or_db(pid)
    return l1[pid]

Reproduce it

Four pods with L1 caches; a price change invalidates L2 only, versus L2 plus a pub/sub broadcast that every pod's L1 subscribes to:

def run(broadcast):
    db = {"price": 100}
    l2 = {"price": 100}
    l1 = {f"pod{i}": {"price": 100} for i in range(1, 5)}   # all pods have it cached
    db["price"] = 80                                         # price change
    l2.pop("price")                                          # DEL in Redis
    if broadcast:
        for pod in l1.values():                              # PUBLISH invalidate rules:p1 → each pod evicts
            pod.pop("price", None)
    served = {}
    for name, cache in l1.items():
        if "price" not in cache:
            cache["price"] = l2.setdefault("price", db["price"])
        served[name] = cache["price"]
    return served

print("L2 delete only:      ", run(False))
print("L2 delete + pub/sub: ", run(True))

L2 delete only:       {'pod1': 100, 'pod2': 100, 'pod3': 100, 'pod4': 100}
L2 delete + pub/sub:  {'pod1': 80, 'pod2': 80, 'pod3': 80, 'pod4': 80}

Root cause

Each pod's L1 is an independent copy. Deleting the shared L2 entry doesn't reach them, so every pod keeps its old value until its own TTL — and different pods expire at different times, which users experience as values flipping back and forth.

The fix

• Broadcast invalidations to all instances (Redis pub/sub, a Kafka topic, or Redis client-side caching with tracking), and evict from L1 on receipt.
• Keep L1 TTLs short (seconds) as the backstop for missed messages (pub/sub is fire-and-forget).
• Cache in L1 only data that tolerates a few seconds of staleness; read critical values from L2/database.

@Bean
RedisMessageListenerContainer invalidations(RedisConnectionFactory f, Cache<Long, PriceRules> l1) {
    var c = new RedisMessageListenerContainer();
    c.setConnectionFactory(f);
    c.addMessageListener((msg, pattern) -> l1.invalidate(Long.parseLong(new String(msg.getBody()))),
                         new ChannelTopic("invalidate:rules"));
    return c;
}
// on update, after commit:
redis.delete("rules:" + productId);
redis.convertAndSend("invalidate:rules", String.valueOf(productId));

def listen_for_invalidations():
    p = r.pubsub()
    p.subscribe("invalidate:rules")
    for msg in p.listen():
        if msg["type"] == "message":
            l1.pop(int(msg["data"]), None)

threading.Thread(target=listen_for_invalidations, daemon=True).start()
# writer: r.delete(f"rules:{pid}"); r.publish("invalidate:rules", pid)

CLIENT TRACKING ON          # Redis 6+: the server sends invalidation messages for keys this client read

Trade-offs

• Pub/sub messages are lost if a pod is disconnected — short L1 TTLs and reconnect-time L1 flushes cover that.
• Every write now fans out to every pod; batch invalidations for bulk updates.
• L1 multiplies memory by the number of pods and can still disagree for a few milliseconds.

Detect it in production

• Version/ETag of the served value per pod in responses or logs; alert when pods disagree for longer than expected.
• Subscriber connection status and invalidation message rates.

Key points

• L1 caches are per-instance copies; deleting L2 doesn't reach them.
• Broadcast invalidations (pub/sub, Kafka, client tracking) and keep L1 TTLs short.
• Use L1 only for data that tolerates brief staleness.