Rate Limits, Retries & Timeouts — Integrating LLM APIs

LLM providers limit requests per minute and input/output tokens per minute per organization or workspace, and may return overloaded errors under load. Handle 429/5xx/529-style errors with exponential backoff + jitter and respect retry-after. Official SDKs already retry a few times — configure max_retries and timeout rather than stacking your own loop on top. Add your own limiter in front (per tenant/user, token-aware) so one customer can't exhaust the shared provider quota, and queue non-interactive work.

Error handling matrix

Error	Retry?	Action
429 rate limit	Yes	Wait retry-after; reduce concurrency
Overloaded / 5xx	Yes	Backoff + jitter; consider fallback
Timeout / connection error	Yes (idempotent)	Retry; stream long outputs
400 invalid request	No	Fix request (context too long, bad params)
401 / 403	No	Credentials/permissions
Refusal (stop reason)	Not blindly	Handle per policy (fallback, explain)

client = anthropic.Anthropic(max_retries=4, timeout=120.0)   # SDK retries 408/409/429/5xx + connection errors

try:
    response = client.messages.create(model=MODEL, max_tokens=4096, messages=messages)
except anthropic.RateLimitError as e:
    retry_after = int(e.response.headers.get("retry-after", "30"))
    raise TryLater(retry_after)                               # e.g. requeue the job
except anthropic.BadRequestError as e:
    log.error("llm_bad_request", message=e.message)           # not retryable
    raise
except anthropic.APIStatusError as e:
    if e.status_code >= 500:
        return fallback_answer()
    raise
except anthropic.APIConnectionError:
    return fallback_answer()

const client = new Anthropic({ maxRetries: 4, timeout: 120_000 });   // TS timeout is milliseconds

try {
  return await client.messages.create({ model: MODEL, max_tokens: 4096, messages });
} catch (e) {
  if (e instanceof Anthropic.RateLimitError) throw new TryLater(Number(e.headers?.get("retry-after") ?? 30));
  if (e instanceof Anthropic.BadRequestError) throw e;            // fix the request
  if (e instanceof Anthropic.APIError && (e.status ?? 0) >= 500) return fallbackAnswer();
  throw e;
}

Common pitfalls

Your retry loop wrapped around the SDK's retries (retry multiplication).
Retrying 400s like 'prompt too long'.
One tenant consuming the whole provider quota.

Key points

Respect retry-after; backoff + jitter
Configure SDK retries and timeouts
Per-tenant token-aware limits in front

Error handling matrix

Common pitfalls

Key points

Practice questions