Stop Reasons, Refusals, Fallbacks & Graceful Degradation — Integrating LLM APIs

Plan for non-happy paths: truncation (max_tokens), refusals (the model declines; check the stop reason before reading content), invalid structured output, provider outages and latency spikes. Strategies: fallback to another model or provider behind an abstraction, cached or rule-based answers, queue-and-notify for async tasks, and honest UX ("AI assistant is unavailable — here's the help center"). Use circuit breakers so a failing provider doesn't consume your timeouts, and feature-flag AI features so you can turn them off.

Failure handling

Failure	Detection	Response
Truncated output	stop_reason = max_tokens	Continue/raise limit; don't use partial JSON
Refusal	stop_reason = refusal	Explain to user; fallback per policy
Invalid output	Validation error	One repair retry, then fallback
Provider 5xx / overload	Errors, latency SLO breach	Circuit breaker → fallback model/provider
Slow responses	p95 latency	Stream; lower effort/model; timeouts
Feature misbehaving	Evals, user reports	Kill switch

Python

def answer(question: str) -> Answer:
    if not flags.enabled("ai_answers") or breaker.is_open():
        return Answer.unavailable()
    try:
        r = client.messages.create(model=MODEL, max_tokens=2048, messages=[...])
    except (anthropic.APIConnectionError, anthropic.InternalServerError):
        breaker.record_failure()
        return Answer.unavailable()
    if r.stop_reason == "refusal":
        return Answer.declined()
    if r.stop_reason == "max_tokens":
        log.warning("llm_truncated", output_tokens=r.usage.output_tokens)
    return Answer.from_message(r)

Common pitfalls

Reading content without checking the stop reason.
No way to disable the AI feature quickly.
Fallback to a model never evaluated on your tasks.

Key points

Branch on stop reasons
Circuit breakers + kill switches
Evaluated fallbacks and honest UX

Failure handling

Common pitfalls

Key points

Practice questions