Database Caching at Scale · Cache-Aside Done Right
Cache-Aside in Practice
The read path, the write path and the details (keys, TTLs, serialization, failures) that decide whether a cache helps or hurts.
Scenario
Example app: an online store's product page reads a product, its price and its stock from Postgres: 3 queries, ~12 ms. The page gets 8,000 requests per second during a sale; 95% of views hit the same 2,000 products.
What you see in production
The team adds Redis. Database load drops — but support reports wrong prices for up to an hour after merchandisers change them, and when Redis restarts the site slows to a crawl. One deploy that changed the Product class broke every product page with deserialization errors.
The naive code
Cache reads with a long TTL and update the cache on write:
def get_product(pid):
data = r.get(f"product:{pid}")
if data:
return pickle.loads(data) # class layout baked into the cache
product = Product.objects.get(pk=pid)
r.set(f"product:{pid}", pickle.dumps(product), ex=3600)
return product
def update_price(pid, price):
Product.objects.filter(pk=pid).update(price=price)
r.set(f"product:{pid}", pickle.dumps(Product.objects.get(pk=pid))) # no TTL now, and racy
@Cacheable(value = "products", key = "#id")
public Product get(long id) { return repo.findById(id).orElseThrow(); }
@Transactional
public void updatePrice(long id, BigDecimal price) {
repo.updatePrice(id, price); // cache never evicted → stale until it expires (or forever)
}
$product = Cache::remember("product:$id", 3600, fn () => Product::findOrFail($id));
// updatePrice(): $product->update(['price' => $p]); ← cache untouched
const hit = await redis.get(`product:${id}`);
if (hit) return JSON.parse(hit);
const product = await prisma.product.findUniqueOrThrow({ where: { id } });
await redis.set(`product:${id}`, JSON.stringify(product)); // no TTL at all
Reproduce it
A cache-aside read path with hit/miss accounting, then a price change that forgets to invalidate:
db = {"p1": {"price": 100}, "p2": {"price": 50}}
cache, stats = {}, {"hit": 0, "miss": 0}
def get(pid):
if pid in cache:
stats["hit"] += 1
return cache[pid]
stats["miss"] += 1
value = dict(db[pid]) # SELECT … (copy, like deserializing)
cache[pid] = value # SET product:{pid} … EX 3600
return value
for pid in ["p1", "p1", "p2", "p1", "p2", "p1"]:
get(pid)
print("after 6 reads:", stats, f"hit ratio {stats['hit'] / 6:.0%}")
db["p1"]["price"] = 80 # merchandiser changes the price; cache not invalidated
print("database price:", db["p1"]["price"], "| page shows:", get("p1")["price"], "<- stale until TTL")
cache.pop("p1") # the fix: DEL product:p1 after the commit
print("after invalidation, page shows:", get("p1")["price"])
Output
after 6 reads: {'hit': 4, 'miss': 2} hit ratio 67%
database price: 80 | page shows: 100 <- stale until TTL
after invalidation, page shows: 80Root cause
Cache-aside (also called look-aside) puts the application in charge of keeping the cache correct. On reads it fills the cache on a miss; on writes it must invalidate whatever the write made wrong. Forgetting the write path — or updating the cache with a value read at the wrong moment — leaves stale data until the TTL. Language-specific serialization (pickle, Java serialization) ties cached bytes to class layouts, so a deploy can make every entry unreadable.
The fix
KEY = "v3:product:{}" # schema version in the key
def get_product(pid):
data = r.get(KEY.format(pid))
if data is not None:
return ProductDTO.model_validate_json(data) # explicit, versioned JSON
product = ProductDTO.from_model(Product.objects.get(pk=pid))
r.set(KEY.format(pid), product.model_dump_json(), ex=300 + random.randint(0, 60))
return product
def update_price(pid, price):
with transaction.atomic():
Product.objects.filter(pk=pid).update(price=price)
transaction.on_commit(lambda: r.delete(KEY.format(pid))) # delete AFTER commit
@Cacheable(value = "products", key = "#id")
public ProductDto get(long id) { return ProductDto.from(repo.findById(id).orElseThrow()); }
@Transactional
@CacheEvict(value = "products", key = "#id") // evicts after the method; with a transaction-aware cache manager, after commit
public void updatePrice(long id, BigDecimal price) { repo.updatePrice(id, price); }
// RedisCacheManager.builder(factory).transactionAware() + JSON serializer + entryTtl(Duration.ofMinutes(5))
$product = Cache::remember("v3:product:$id", now()->addMinutes(5), fn () => Product::findOrFail($id)->toArray());
DB::transaction(function () use ($id, $price) {
Product::whereKey($id)->update(['price' => $price]);
DB::afterCommit(fn () => Cache::forget("v3:product:$id"));
});
const key = `v3:product:${id}`;
const hit = await redis.get(key);
if (hit !== null) return JSON.parse(hit) as ProductDto;
const product = toDto(await prisma.product.findUniqueOrThrow({ where: { id } }));
await redis.set(key, JSON.stringify(product), "EX", 300 + Math.floor(Math.random() * 60));
return product;
// on update: after the transaction resolves → await redis.del(key)
Rules of thumb:
- Invalidate (delete) on write, after commit — don't compute and set the new value in the write path. Facebook's memcache paper describes the same choice: "We choose to delete cached data instead of updating it because deletes are idempotent" (Scaling Memcache at Facebook).
- Always set a TTL — it bounds the damage of every missed invalidation.
- Cache DTOs as JSON (or another explicit format) with a version in the key.
- Treat the cache as optional: on a Redis error, fall back to the database (with protection — see stampedes).
Trade-offs
- Cache-aside caches only what's read (memory-efficient) but the first read after each invalidation is slow.
- Short TTLs reduce staleness but increase misses; pick per data type (prices minutes, product descriptions hours).
- Caching whole ORM objects is convenient but brittle; cache the shape the page needs.
Detect it in production
- Hit ratio per key prefix, cache latency, and database QPS from cache misses.
- "Staleness probes": periodically compare a sample of cached values with the database.
- Deserialization error counts after deploys.
Key points
- Cache-aside: read → miss → load → set with TTL; write → commit → delete.
- Delete instead of update; always use TTLs; version keys and use explicit serialization.
- The cache must be optional and its correctness monitored.
Practice questions
Why invalidate (delete) the cache entry on write instead of setting the new value?
Answer
Deletes are idempotent and can't write an older value over a newer one; the next read loads the current row. Setting values from the write path can race with other writers and readers and leave stale data.
Why should every cache entry have a TTL even if you invalidate on write?
Answer
Invalidations get missed (bugs, crashes, network errors, new code paths). A TTL bounds how long a wrong value can live.
A deploy changed a class and every product page started failing. What caching practice prevents this?
Answer
Cache explicit, versioned representations (e.g. JSON DTOs) and include a schema version in the key, so new code reads new keys instead of deserializing old binary objects.
Open in the app to mark lessons complete, take notes and track your progress.