Hedged Requests and Tail Latency — Resilience Patterns

In fan-out systems the slowest backend dominates: if a page calls 50 backends with 1% slow responses each, ~40% of pages hit at least one slow call. Hedged requests send a second copy of an idempotent request to another replica if the first hasn't answered by roughly the p95, using whichever returns first (then cancelling the other). Limit hedges to a small percentage of traffic, and only for idempotent reads.

Tail-latency techniques

Technique	Reduces	Cost
Hedged requests	Tail latency	Extra load (a few %)
Tied requests (cancel on start)	Tail latency	Lower extra load; more complex
Timeouts + partial results	Worst-case latency	Incomplete responses
Reducing fan-out	Probability of slow calls	Design work

Go (hedged read)

func hedgedGet(ctx context.Context, replicas []string, key string, hedgeAfter time.Duration) ([]byte, error) {
    ctx, cancel := context.WithCancel(ctx)
    defer cancel()                                   // cancels the slower request
    results := make(chan result, 2)
    go fetch(ctx, replicas[0], key, results)
    select {
    case r := <-results:
        return r.body, r.err
    case <-time.After(hedgeAfter):                   // e.g. observed p95
        go fetch(ctx, replicas[1], key, results)
    }
    r := <-results
    return r.body, r.err
}

Common pitfalls

Hedging non-idempotent writes.
Hedging every request immediately (doubles load).
Not cancelling the losing request.

Key points

Fan-out amplifies tail latency
Hedge idempotent reads after ~p95
Cap hedging traffic

Further reading: Dean & Barroso: The Tail at Scale

Practice questions

Why does fan-out make tail latency worse?

Answer

A request's latency is the maximum of its sub-requests; with many sub-requests, the chance that at least one hits a rare slow path becomes high.

When is it safe to hedge a request?

Answer

When it's idempotent (typically reads) and the extra load is acceptable; writes would risk duplicate side effects.