Terraform in Production · Capstone Case Studies

Capstone: Postmortem — The Plan That Deleted a Queue

Reconstruct an incident from a plan, a pipeline and a timeline; find the root causes and write the remediation plan.

Scenario

Example company: an online marketplace. On Monday at 16:05 the orders-events SQS queue in production was deleted and recreated by Terraform. About 38,000 in-flight messages were lost; order confirmations and seller payouts for those orders had to be reconstructed from the database over two days.

Evidence

PR #1432, titled "Tidy up queue naming", changed count = length(var.queues) resources: it sorted var.queues alphabetically and removed an unused queue orders-archive from the middle of the list.
The PR plan comment was 900 lines (tag updates on 120 resources). The summary line said Plan: 3 to add, 121 to change, 3 to destroy.
The pipeline: plan on PR; on merge, terraform apply -auto-approve (fresh plan) with an IAM user's access key stored in CI secrets.
15:40: PR #1430 (unrelated) merged and applied in the same state.
aws_sqs_queue had no prevent_destroy; there is no CI check on plan contents.
The queue's consumers were paused at 16:05 by a separate alarm, and messages on the old queue were gone once it was deleted.

Deliverables

The timeline and the technical root cause.
Contributing factors in the process.
Immediate fixes (this week) and structural fixes (this quarter).
How you'd confirm the fix works.

Draft each deliverable before revealing the model answer and rubric.

Further reading: for_each · Refactoring with moved blocks · JSON plan format

Deliverables

What's the technical root cause?

Model answer

The queues used count over a list. Sorting the list and removing orders-archive changed which name sat at each index. Because an SQS queue's name forces replacement, every index whose name changed was planned as destroy and create, including orders-events. The "3 to destroy" in the summary were replacements of live queues, not just the removal of the unused one.

Rubric

Full marks: identifies index-based identity of count plus the force-new name argument as the mechanism, and that the summary hid replacements.

List the contributing factors in the process.

Model answer

The reviewed plan was not the applied plan (apply -auto-approve on merge re-planned after #1430).
Destructive changes were hidden in a 900-line plan with no automated summary or check.
A cosmetic refactor was mixed with tag changes, so a large plan looked normal.
No prevent_destroy/protection on stateful queues; no dead-letter or replay path for in-flight messages.
Long-lived CI access keys with broad permissions (not causal, but it widened the potential damage).

Rubric

Full marks: at least four factors spanning review, pipeline, protection and change hygiene. Avoid blaming the individual engineer.

What are the immediate and structural fixes?

Model answer

This week: convert the queues to for_each keyed by name with moved blocks from each old index (the plan must be all moves); add lifecycle { prevent_destroy = true } to stateful queues; add a CI plan guard that fails on delete for aws_sqs_queue, aws_db_instance, aws_s3_bucket and similar without an allow-destroy label.

This quarter: apply saved plan files only; merge queue so plans are current; destructive-actions summary posted on every PR; split service stacks from shared infrastructure; OIDC roles instead of access keys; a rule that refactor PRs must plan to zero changes; message durability (producers can replay from the outbox table).

Rubric

Full marks: fixes the identity bug with moved blocks (not by recreating), adds both a Terraform-level and a CI-level guard, and fixes plan-vs-apply mismatch.

How would you confirm the fix works?

Model answer

Replay the incident: in a sandbox account, apply the old config, then the #1432 change, and confirm the CI guard blocks it and prevent_destroy errors.
Check the migration PR's plan: only has moved to lines, 0 to add, 0 to change, 0 to destroy.
Track "PRs blocked by the destroy guard" and "plans applied from saved files" (should be 100%) for a month.

Rubric

Full marks: tests the guard against the actual failure scenario and defines ongoing measures.

Open in the app to mark lessons complete, take notes and track your progress.