Concurrency Limits and Adaptive Limiting — Rate Limiting, Quotas & Abuse Protection

Requests-per-second limits don't capture cost: 10 slow exports can saturate a service. Concurrency limits cap in-flight requests per client or endpoint (semaphores). Adaptive concurrency (Netflix concurrency-limits, Envoy adaptive concurrency) adjusts limits automatically using latency measurements (Little's Law, gradient algorithms), shedding load before queues explode.

Limiting by rate vs concurrency

Criterion	Rate limit	Static concurrency limit	Adaptive concurrency
Measures	Arrivals per time	In-flight work	In-flight vs latency
Handles slow requests	No	Yes	Yes
Tuning	Manual	Manual	Automatic
Example	1000/min	5 exports at once	Envoy adaptive_concurrency

type ConcurrencyLimiter struct{ mu sync.Mutex; inFlight map[string]int; max int }

func (l *ConcurrencyLimiter) Acquire(key string) bool {
    l.mu.Lock(); defer l.mu.Unlock()
    if l.inFlight[key] >= l.max { return false }
    l.inFlight[key]++
    return true
}
func (l *ConcurrencyLimiter) Release(key string) {
    l.mu.Lock(); defer l.mu.Unlock()
    l.inFlight[key]--
}
// handler: if !lim.Acquire(apiKey) { 429 }; defer lim.Release(apiKey)

from collections import defaultdict
import asyncio

semaphores: dict[str, asyncio.Semaphore] = defaultdict(lambda: asyncio.Semaphore(5))

async def run_export(api_key: str, job):
    sem = semaphores[api_key]
    if sem.locked():
        raise TooManyConcurrent()        # → 429 with Retry-After
    async with sem:
        return await job()

Common pitfalls

Rate-only limits on long-running endpoints.
Per-process semaphores assumed to be global.
Releasing permits only on success (leaks on exceptions).

Key points

Concurrency limits for expensive work
Adaptive limits react to latency
Always release in finally/defer

Further reading: Netflix concurrency-limits

Practice questions

Why can a service with a generous rate limit still be overloaded by a single client?

Answer

If each request is slow or expensive, even a low rate accumulates many concurrent requests that exhaust workers or database connections.

How do adaptive concurrency limiters decide the limit?

Answer

They track latency (e.g., minimum RTT vs current RTT) and increase the allowed concurrency while latency stays low, decreasing it when latency rises, following queueing theory like Little's Law.