Managing Connections at Scale — Real-Time APIs

Persistent connections change operations: load balancers must support WebSockets/long-lived HTTP and idle timeouts above heartbeat intervals; connection draining during deploys (stop accepting, send a reconnect hint, close gradually); jittered reconnects on clients to avoid storms; limits per user/IP; and capacity planning by concurrent connections (memory per connection, file descriptors) rather than requests per second.

Operational practices

Concern	Practice
Load balancer	WebSocket support; idle timeout > heartbeat interval
Deploys	Drain: stop accepting, send 'reconnect' message, close over minutes
Reconnect storms	Client exponential backoff with jitter; server admission control
Capacity	Plan by concurrent connections (memory, FDs), not RPS
Abuse	Max connections per user/IP; message rate limits

# Kubernetes: give WebSocket pods time to drain on deploys
terminationGracePeriodSeconds: 300
lifecycle:
  preStop:
    exec: { command: ["/bin/sh", "-c", "kill -USR1 1 && sleep 240"] }   # app: stop accepting, notify clients, close gradually

Common pitfalls

Killing all connections at once during deploys.
LB idle timeouts shorter than heartbeats.
No per-user connection limits.

Key points

Drain gracefully
Jittered client reconnects
Plan capacity by connections

Further reading: RFC 6455: The WebSocket Protocol

Practice questions

Why can a deploy cause an outage for WebSocket services?

Answer

Restarting all pods at once disconnects every client; they reconnect simultaneously (reconnect storm), overwhelming auth and gateways.

What resource usually limits the number of WebSocket connections per server?

Answer

Memory per connection (buffers, state) and file descriptors, rather than CPU or request rate.