Database Caching at Scale · Operating Caches in Production
Redis Data Structures and Atomic Operations
Use INCR, hashes, sorted sets and Lua scripts instead of racy GET-then-SET sequences.
Scenario
Example app: a game backend keeps per-player daily limits ("max 5 free spins per day") and a leaderboard in Redis. The limit check reads the counter, compares it with 5, then writes the incremented value.
What you see in production
Players with fast fingers (or scripts) get 7–9 free spins a day. The leaderboard, stored as one JSON blob per game, is rewritten on every score update, and concurrent updates occasionally drop scores.
The naive code
def try_spin(player):
used = int(r.get(f"spins:{player}") or 0) # read
if used >= 5:
return False
r.set(f"spins:{player}", used + 1, ex=86400) # write: concurrent requests both pass the check
return True
def add_score(game, player, score):
board = json.loads(r.get(f"board:{game}") or "{}") # read-modify-write of a whole JSON blob
board[player] = max(board.get(player, 0), score)
r.set(f"board:{game}", json.dumps(board))
Reproduce it
Two requests interleave a GET-then-SET counter; then the same with an atomic increment-and-check (what a Lua script does inside Redis):
def racy(store):
def spin():
used = store.get("spins", 0) # GET
yield
if used < 5:
store["spins"] = used + 1 # SET
yield "allowed"
else:
yield "denied"
return spin
def atomic(store):
def spin():
yield # the whole script runs as one step inside Redis
if store.get("spins", 0) < 5:
store["spins"] = store.get("spins", 0) + 1
yield "allowed"
else:
yield "denied"
return spin
for name, factory in (("GET then SET", racy), ("atomic script", atomic)):
store = {"spins": 4} # one free spin left
spin = factory(store)
a, b = spin(), spin()
next(a); next(b) # both start
results = [next(a), next(b)]
print(f"{name:13} -> {results}, spins used = {store['spins']}")
Output
GET then SET -> ['allowed', 'allowed'], spins used = 5 atomic script -> ['allowed', 'denied'], spins used = 5
Root cause
Redis executes each command atomically, but a sequence of commands from a client is not atomic: other clients' commands interleave between your GET and SET. Storing whole aggregates as one blob also turns every small update into a read-modify-write of the entire value.
The fix
Use the data structure that makes the update a single command, or run the logic server-side in a script:
-- KEYS[1] = spins:{player}, ARGV[1] = limit, ARGV[2] = ttl
local used = tonumber(redis.call('GET', KEYS[1]) or '0')
if used >= tonumber(ARGV[1]) then return 0 end
redis.call('INCR', KEYS[1])
if used == 0 then redis.call('EXPIRE', KEYS[1], ARGV[2]) end
return 1
try_spin_script = r.register_script(LUA_TRY_SPIN)
allowed = try_spin_script(keys=[f"spins:{player}"], args=[5, 86400]) == 1
# leaderboard: a sorted set; GT only raises scores (Redis 6.2+)
r.zadd(f"board:{game}", {player: score}, gt=True)
top10 = r.zrevrange(f"board:{game}", 0, 9, withscores=True)
RedisScript<Long> trySpin = RedisScript.of(new ClassPathResource("try_spin.lua"), Long.class);
boolean allowed = redis.execute(trySpin, List.of("spins:" + player), "5", "86400") == 1L;
redis.opsForZSet().add("board:" + game, player, score);
$allowed = Redis::eval($tryspinLua, 1, "spins:$player", 5, 86400) === 1;
Redis::zadd("board:$game", 'GT', $score, $player);
redis.defineCommand("trySpin", { numberOfKeys: 1, lua: TRY_SPIN_LUA });
const allowed = (await (redis as any).trySpin(`spins:${player}`, 5, 86400)) === 1;
await redis.zadd(`board:${game}`, "GT", score, player);
Trade-offs
- Lua scripts block Redis while they run — keep them short and O(1)/O(log n).
- In Redis Cluster, all keys a script touches must be in the same hash slot (use hash tags like
{player42}:spins). MULTI/EXECtransactions batch commands atomically but can't branch on values read inside them; useWATCHfor optimistic checks or a script.
Detect it in production
- Business invariants: count spins per player per day in the database/event log and compare with the limit.
- Redis
SLOWLOGfor long scripts; latency monitoring.
Key points
- Single Redis commands are atomic; sequences aren't.
- Prefer native structures (INCR, hashes, sorted sets) and server-side scripts for check-and-set logic.
- Keep scripts short; respect hash slots in clusters.
Practice questions
Why does a GET-then-SET limit check allow more than the limit under concurrency?
Answer
Other clients' commands interleave between the GET and the SET; two requests can both read 4 and both write 5, so both pass the check.
How do you store a leaderboard in Redis without read-modify-write races?
Answer
Use a sorted set: ZADD (optionally with GT to only raise scores) updates one member atomically, and ZREVRANGE returns the top N.
What constraint applies to Lua scripts in Redis Cluster?
Answer
All keys accessed by the script must hash to the same slot; use hash tags ({...}) to co-locate them.
Open in the app to mark lessons complete, take notes and track your progress.