Safe Deployments: Versions, Aliases and Canaries

Scenario

Example app: the checkout function is deployed by updating $LATEST directly from CI, and API Gateway invokes $LATEST.

What you see in production

A dependency upgrade throws on a rare input format. Because the change goes live for 100% of traffic instantly, checkout fails for every affected request until someone notices, finds the previous build and redeploys it, about 25 minutes later.

The naive code

aws lambda update-function-code --function-name checkout --zip-file fileb://dist/checkout.zip
# API Gateway integration points at the unqualified function ($LATEST)

Reproduce it

Compare an all-at-once release with a 10% canary that rolls back automatically when the error-rate alarm fires:

const requestsPerMinute = 1000, newVersionErrorRate = 0.08, alarmThreshold = 0.02;

function allAtOnce(minutesToNotice) {
  return Math.round(requestsPerMinute * newVersionErrorRate * minutesToNotice);
}
function canary(canaryShare, alarmEvaluationMinutes) {
  let failed = 0;
  for (let minute = 1; minute <= alarmEvaluationMinutes; minute++) {
    const canaryRequests = requestsPerMinute * canaryShare;
    failed += canaryRequests * newVersionErrorRate;
    if (newVersionErrorRate > alarmThreshold && minute === alarmEvaluationMinutes) return { failed: Math.round(failed), result: "alarm -> rolled back" };
  }
  return { failed: Math.round(failed), result: "promoted" };
}
console.log("all at once, noticed after 25 min:", allAtOnce(25), "failed requests");
console.log("canary 10%, 3-minute alarm:      ", canary(0.1, 3));

all at once, noticed after 25 min: 2000 failed requests
canary 10%, 3-minute alarm:       { failed: 24, result: 'alarm -> rolled back' }

Root cause

Deploying straight to $LATEST gives no gradual exposure, no automatic health check and no instant rollback target. The blast radius of every bad change is all traffic, and recovery depends on a human noticing and redeploying.

The fix

Publish an immutable version for every deployment and point callers at an alias (for example live). Shift traffic between versions gradually with CodeDeploy, driven by alarms, and keep the previous version for instant rollback. With AWS SAM:

Checkout:
  Type: AWS::Serverless::Function
  Properties:
    AutoPublishAlias: live
    DeploymentPreference:
      Type: Canary10Percent5Minutes          # or Linear10PercentEvery1Minute, AllAtOnce
      Alarms: [!Ref CheckoutErrorRateAlarm, !Ref CheckoutLatencyAlarm]
      Hooks:
        PreTraffic: !Ref CheckoutSmokeTest    # run tests against the new version before shifting traffic

CDK (LambdaDeploymentGroup) and Terraform (with CodeDeploy resources or weighted aliases) support the same pattern. Infrastructure as code for everything means a release is a reviewed commit, and rollback is redeploying the previous one.

Trade-offs

• Canaries need enough traffic during the bake time to trigger alarms; low-traffic functions need synthetic tests or longer windows.
• Two versions run side by side: changes to events, databases or APIs must be backward compatible.
• Provisioned concurrency is configured per version or alias, so plan it with deployments.

Detect it in production

• Alarms on error rate and latency per alias and version, wired to CodeDeploy.
• Deployment markers on dashboards to correlate regressions with releases.
• Track how often deployments roll back automatically (a healthy signal, not a failure).

Key points

• Never deploy straight to $LATEST for production traffic.
• Versions + aliases + CodeDeploy canaries with alarm-based rollback.
• Keep changes backward compatible while versions overlap.

Further reading: Lambda function aliases · Deploying serverless applications gradually with AWS SAM