Configuration and Secrets

Scenario

Example app: a payments webhook function needs a database password and a provider API key. A security review asked the team to move them out of environment variables, so they now read both from AWS Secrets Manager.

What you see in production

During a sale the function starts failing with throttling errors from Secrets Manager, latency rises by 80 ms per request, and the monthly bill shows millions of secret retrievals. When a password is rotated, some invocations fail for several minutes.

The naive code

export const handler = async (event) => {
  const secret = JSON.parse((await secrets.getSecretValue({ SecretId: "prod/payments" })).SecretString);
  const db = await connect({ password: secret.dbPassword });
  // ...
};

Reproduce it

Compare calls to the secrets service for 10,000 invocations across 20 execution environments, with no cache and with a 5-minute in-memory cache:

function simulate({ cacheTtlSeconds }) {
  const invocations = 10000, environments = 20, durationSeconds = 600;
  let calls = 0;
  const cache = new Map();
  for (let i = 0; i < invocations; i++) {
    const env = i % environments;
    const now = Math.floor((i / invocations) * durationSeconds);
    const hit = cache.get(env);
    if (cacheTtlSeconds && hit && now - hit < cacheTtlSeconds) continue;
    calls++;
    cache.set(env, now);
  }
  return calls;
}
console.log("no cache:       ", simulate({ cacheTtlSeconds: 0 }), "secret API calls");
console.log("5-minute cache: ", simulate({ cacheTtlSeconds: 300 }), "secret API calls");

no cache:        10000 secret API calls
5-minute cache:  57 secret API calls

Root cause

Secret managers are remote APIs with request quotas, latency and per-call cost. Calling them on every invocation turns a configuration lookup into a hot-path dependency that scales with traffic. Environment variables avoid the calls but are visible to anyone who can read the function configuration and change only when the function is redeployed or updated.

The fix

Cache configuration per execution environment, with a sensible TTL, using a library or the managed extension:

import { getSecret } from "@aws-lambda-powertools/parameters/secrets";

export const handler = async (event) => {
  const secret = await getSecret("prod/payments", { transform: "json", maxAge: 300 }); // cached for 5 minutes
  // ...
};

Alternatives: the AWS Parameters and Secrets Lambda Extension, which serves cached values from a local HTTP endpoint, or loading once at init with top-level await when values rarely change. Keep non-sensitive configuration in environment variables or Parameter Store; keep secrets in Secrets Manager with least-privilege access (the function role can read only its own secrets). For rotation, make clients retry once with a fresh secret on authentication errors, and use rotation strategies that keep the previous credential valid briefly.

Trade-offs

• Longer cache TTLs mean fewer calls but slower pickup of rotated values; pair a TTL with refresh-on-auth-failure.
• Loading at init speeds every request but makes rotation require new environments.
• Environment variables are simplest and fine for non-secret configuration; they are encrypted at rest but readable by anyone with permission to view the function.

Detect it in production

• CloudWatch metrics and CloudTrail events for GetSecretValue volume and throttling.
• Latency traces (X-Ray, OpenTelemetry) showing a secrets call in every request.
• Authentication failures clustered around rotation times.

Key points

• Never fetch secrets on every invocation; cache per environment with a TTL.
• Use Powertools Parameters or the Parameters and Secrets extension.
• Non-secret config in environment variables or Parameter Store; secrets in Secrets Manager.
• Handle rotation with refresh-on-failure.

Further reading: Using Secrets Manager secrets in Lambda functions · Powertools for AWS Lambda (TypeScript)