Load Shedding and Admission Control — Resilience Patterns

When demand exceeds capacity, a server that accepts everything slows down for everyone and eventually collapses. Load shedding rejects some requests quickly (503 + Retry-After) to keep goodput high. Shed by priority (health checks and checkout before recommendations and batch), use admission control based on concurrency or queue time, drop requests whose deadline has already passed, and prefer cheap rejection at the edge.

Shedding strategies

Strategy	Signal	Pros	Cons
Concurrency limit	In-flight requests	Simple, effective	Needs tuning
Queue time	Time waiting before processing	Drops doomed work	Needs timestamp
CPU/latency-based	Utilization or p99	Adaptive	Can oscillate
Priority shedding	Request class	Protects critical paths	Requires classification

var inflight atomic.Int64
const maxInflight = 500

func shed(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        critical := r.URL.Path == "/healthz" || strings.HasPrefix(r.URL.Path, "/checkout")
        if n := inflight.Add(1); n > maxInflight && !critical {
            inflight.Add(-1)
            w.Header().Set("Retry-After", "2")
            http.Error(w, "overloaded", http.StatusServiceUnavailable)
            return
        }
        defer inflight.Add(-1)
        next.ServeHTTP(w, r)
    })
}

@app.middleware("http")
async def shed_stale(request: Request, call_next):
    # Gateway sets X-Request-Start (ms); drop work that already waited too long
    started = float(request.headers.get("x-request-start", "0")) / 1000
    if started and time.time() - started > 2.0 and request.url.path != "/healthz":
        return JSONResponse({"title": "Overloaded"}, status_code=503, headers={"Retry-After": "2"})
    return await call_next(request)

Common pitfalls

Shedding health checks (orchestrator kills healthy-but-busy pods).
Expensive rejection paths.
No Retry-After, so clients retry immediately.

Key points

Reject early and cheaply
Shed by priority
Drop work past its deadline

Further reading: AWS Builders' Library: Load shedding

Practice questions

What is 'goodput' and why does load shedding improve it?

Answer

Goodput is the rate of successfully completed useful requests; without shedding, overload makes nearly all requests time out, so goodput drops toward zero.

Why drop requests that waited too long in a queue?

Answer

Their clients have likely timed out, so processing them wastes capacity that could serve requests that can still succeed.