Concurrency, Throttling and Noisy Neighbours

Scenario

Example app: one AWS account runs a checkout API function and a nightly re-indexing function that fans out over SQS. The account's concurrency quota is the default for the Region.

What you see in production

At 02:00 every night the re-indexing job runs and the checkout API returns 429 Too Many Requests for several minutes. Neither function has errors in its code.

The naive code

No concurrency configuration: all functions share the account's unreserved pool.

Reindex:
  Type: AWS::Serverless::Function
  Properties:
    Events:
      Queue: { Type: SQS, Properties: { Queue: !GetAtt ReindexQueue.Arn, BatchSize: 10 } }

Reproduce it

const accountLimit = 1000;
function minute({ checkoutDemand, reindexDemand, reservedCheckout = 0, reindexMax = Infinity }) {
  const unreserved = accountLimit - reservedCheckout;
  const reindex = Math.min(reindexDemand, reindexMax, unreserved);
  const checkoutCapacity = reservedCheckout || (unreserved - reindex);
  const checkoutServed = Math.min(checkoutDemand, checkoutCapacity);
  return { reindex, checkoutServed, checkoutThrottled: checkoutDemand - checkoutServed };
}
const demand = { checkoutDemand: 180, reindexDemand: 1500 };
console.log("shared pool:                ", minute(demand));
console.log("reserve 250 for checkout:   ", minute({ ...demand, reservedCheckout: 250 }));
console.log("+ cap reindex consumers 300:", minute({ ...demand, reservedCheckout: 250, reindexMax: 300 }));

shared pool:                 { reindex: 1000, checkoutServed: 0, checkoutThrottled: 180 }
reserve 250 for checkout:    { reindex: 750, checkoutServed: 180, checkoutThrottled: 0 }
+ cap reindex consumers 300: { reindex: 300, checkoutServed: 180, checkoutThrottled: 0 }

Root cause

Concurrent executions in a Region share an account-level quota. A function that scales aggressively (here, many SQS batches in parallel) can consume the shared pool, and every other function in the account is throttled. Synchronous callers then receive 429s; asynchronous invocations are retried by Lambda; event source mappings back off.

The fix

• Give critical functions reserved concurrency: guaranteed capacity (and a cap) carved out of the account pool.
• Limit background consumers with the SQS event source mapping's maximum concurrency, instead of letting them take everything.
• Use provisioned concurrency where you also need pre-initialised environments (cold-start-free latency).
• Request quota increases ahead of known peaks, and isolate workloads into separate accounts (AWS Organizations) so environments and teams do not share one pool.

Checkout:
  Type: AWS::Serverless::Function
  Properties:
    ReservedConcurrentExecutions: 250
Reindex:
  Type: AWS::Serverless::Function
  Properties:
    Events:
      Queue:
        Type: SQS
        Properties: { Queue: !GetAtt ReindexQueue.Arn, BatchSize: 10, ScalingConfig: { MaximumConcurrency: 300 } }

Trade-offs

• Reserved concurrency is also a hard cap: set it too low and the critical function throttles itself.
• Capping background work makes it take longer; schedule and size it accordingly.
• Separate accounts add management overhead but give the strongest isolation (quotas, blast radius, permissions).

Detect it in production

• Throttles per function and ConcurrentExecutions / UnreservedConcurrentExecutions for the account.
• Alarms when account concurrency approaches the quota.
• Correlate 429s in API metrics with scheduled jobs.

Key points

• All functions in a Region share the account's concurrency quota.
• Reserve concurrency for critical paths; cap background consumers.
• Provisioned concurrency for latency; separate accounts for isolation.

Further reading: Understanding Lambda function scaling · Configuring reserved concurrency