Streaming Responses to Users — Integrating LLM APIs

Generation takes seconds; streaming shows text as it's produced (the "typing" effect) and avoids HTTP timeouts on long outputs. Your backend consumes the provider stream and relays it to clients — typically via Server-Sent Events (one-way, auto-reconnect, works over HTTP/2). Handle client disconnects (cancel the upstream request to stop paying), send heartbeats through proxies, disable proxy buffering, and persist the final message (with usage) when the stream completes.

Streaming vs not

Criterion	Non-streaming	Streaming (SSE relay)
Time to first token shown	Full generation time	Sub-second
Long outputs	Risk of HTTP timeouts	Safe
Implementation	Simple	Relay, cancellation, partial state
Post-processing (validation)	Easy on full text	After completion (or incremental)
Best for	Background jobs, short outputs	Chat UIs, long generations

from fastapi.responses import StreamingResponse
import anthropic, json

aclient = anthropic.AsyncAnthropic()

@router.post("/v1/chat/stream")
async def chat_stream(body: ChatIn, request: Request, user=Depends(current_user)):
    async def events():
        async with aclient.messages.stream(model=MODEL, max_tokens=64000,
                                           system=SYSTEM, messages=body.messages) as stream:
            async for text in stream.text_stream:
                if await request.is_disconnected():
                    break                                  # leaving the context closes the upstream stream
                yield f"data: {json.dumps({'delta': text})}\n\n"
            final = await stream.get_final_message()
            await save_turn(user.id, final)                # persist text + usage
            yield f"event: done\ndata: {json.dumps({'stop_reason': final.stop_reason})}\n\n"
    return StreamingResponse(events(), media_type="text/event-stream",
                             headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"})

app.post("/v1/chat/stream", requireAuth, async (req, res) => {
  res.writeHead(200, { "Content-Type": "text/event-stream", "Cache-Control": "no-cache", "X-Accel-Buffering": "no" });
  const stream = client.messages.stream({ model: MODEL, max_tokens: 64000, system: SYSTEM, messages: req.body.messages });
  req.on("close", () => stream.abort());                    // stop paying when the user leaves
  for await (const event of stream) {
    if (event.type === "content_block_delta" && event.delta.type === "text_delta") {
      res.write(`data: ${JSON.stringify({ delta: event.delta.text })}\n\n`);
    }
  }
  const final = await stream.finalMessage();
  await saveTurn(req.user.id, final);
  res.end(`event: done\ndata: ${JSON.stringify({ stop_reason: final.stop_reason })}\n\n`);
});

Common pitfalls

Proxy buffering (Nginx) that delivers the whole response at the end.
Not cancelling upstream on disconnect.
Rendering partial Markdown/HTML unsafely as it streams.

Key points

Relay via SSE
Cancel upstream on disconnect
Persist the final message + usage

Streaming vs not

Common pitfalls

Key points

Practice questions