Guard Rails for Destructive Plans

Scenario

Example team: a junior engineer changes the identifier of the orders database to follow a new naming standard. The plan is 600 lines long because of tag updates on dozens of resources. CI applies automatically after one approval.

What you see in production

Buried in the plan: # aws_db_instance.orders must be replaced. The approval comes in, the database is destroyed and a new empty one is created. Restoring from the latest snapshot takes four hours and loses 40 minutes of orders.

The naive code

resource "aws_db_instance" "orders" {
  identifier = "payments-orders-prod"   # was "orders-prod": changing it forces replacement
  # ...
}

and a pipeline that runs terraform apply -auto-approve after a single approval, with nothing reading the plan except a human.

Reproduce it

prevent_destroy makes Terraform refuse any plan that would destroy the resource, including a replacement:

resource "terraform_data" "orders_db" {
  triggers_replace = "orders-prod"
  lifecycle {
    prevent_destroy = true
  }
}
# after: rename the database identifier
resource "terraform_data" "orders_db" {
  triggers_replace = "payments-orders-prod"
  lifecycle {
    prevent_destroy = true
  }
}

Error: Instance cannot be destroyed

  on main.tf line 1:
   1: resource "terraform_data" "orders_db" {

Resource terraform_data.orders_db has lifecycle.prevent_destroy set, but the
plan calls for this resource to be destroyed. To avoid this error and
continue with the plan, either disable lifecycle.prevent_destroy or reduce
the scope of the plan using the -target option.

But it only protects while the block is in the configuration. Delete the block and the protection goes with it:

resource "terraform_data" "orders_db" {
  triggers_replace = "orders-prod"
  lifecycle {
    prevent_destroy = true
  }
}
# after: the whole block deleted in a cleanup PR

Terraform will perform the following actions:

  # terraform_data.orders_db will be destroyed
  # (because terraform_data.orders_db is not in configuration)
  - resource "terraform_data" "orders_db" {
      - id               = "8f4a0616-33d9-97a7-f53f-15e8d97fc1d4" -> null
      - triggers_replace = "orders-prod" -> null
    }

Plan: 0 to add, 0 to change, 1 to destroy.

Root cause

Replacements hide in long plans, and humans approve diffs they don't read. Every safety net here is partial on its own: prevent_destroy disappears with the block, provider-side deletion protection covers only some resource types, and reviewers get tired. You need layers, including one that reads the plan mechanically.

The fix

Layer the protections:

1. Cloud-side deletion protection on stateful resources (survives anything Terraform does):

resource "aws_db_instance" "orders" {
  # ...
  deletion_protection       = true
  skip_final_snapshot       = false
  final_snapshot_identifier = "orders-prod-final"
  lifecycle {
    prevent_destroy = true
  }
}

resource "azurerm_management_lock" "orders_db" {
  name       = "orders-db-no-delete"
  scope      = azurerm_postgresql_flexible_server.orders.id
  lock_level = "CanNotDelete"
}

resource "google_sql_database_instance" "orders" {
  # ...
  deletion_protection = true   # Terraform-side check
  settings {
    deletion_protection_enabled = true   # Cloud SQL API-side protection
  }
}

2. An automated plan check in CI that fails on deletes or replacements of protected types unless the PR carries an explicit label:

import json

# terraform show -json tfplan  (trimmed to the fields the guard reads)
plan = json.loads('''{"resource_changes": [
  {"address": "aws_s3_bucket_policy.logs",  "type": "aws_s3_bucket_policy", "change": {"actions": ["update"]}},
  {"address": "aws_db_instance.orders",     "type": "aws_db_instance",      "change": {"actions": ["delete", "create"]}},
  {"address": "aws_sqs_queue.dead_letter",  "type": "aws_sqs_queue",        "change": {"actions": ["delete"]}},
  {"address": "aws_instance.worker[0]",     "type": "aws_instance",         "change": {"actions": ["create", "delete"]}}
]}''')

PROTECTED = {"aws_db_instance", "aws_rds_cluster", "aws_s3_bucket", "aws_dynamodb_table", "aws_sqs_queue",
             "aws_efs_file_system", "aws_kms_key"}
allow_destroy = False   # set from a PR label such as "allow-destroy"

problems = []
for rc in plan["resource_changes"]:
    actions = rc["change"]["actions"]
    if "delete" not in actions:
        continue
    kind = "replace" if "create" in actions else "delete"
    order = " (create first)" if actions == ["create", "delete"] else ""
    flag = "BLOCK" if rc["type"] in PROTECTED and not allow_destroy else "warn "
    print(f"{flag} {kind:7} {rc['address']}{order}")
    if flag == "BLOCK":
        problems.append(rc["address"])

print(f"{len(problems)} protected resource(s) would be destroyed" if problems else "plan is safe")
raise SystemExit(0)   # in CI: raise SystemExit(1 if problems else 0)

BLOCK replace aws_db_instance.orders
BLOCK delete  aws_sqs_queue.dead_letter
warn  replace aws_instance.worker[0] (create first)
2 protected resource(s) would be destroyed

3. Apply the reviewed plan file and require approval from an owner of the affected stack, with the destructive-action summary posted as a PR comment so nobody has to find it in 600 lines.

For resources that can be replaced but shouldn't go down while that happens, use create_before_destroy = true (and a name_prefix instead of a fixed name, so old and new can exist at the same time).

Trade-offs

• Deletion protection makes intentional teardown a two-step change (turn protection off, then delete). That friction is the point for production data.
• Plan guards need a curated list of protected types and an escape hatch (a label plus a second approver).
• create_before_destroy needs capacity, quota and unique names for two copies at once.

Detect it in production

• Post a "destructive actions" summary on every PR, even when CI passes.
• Alert on delete API calls for databases, buckets and keys (CloudTrail, Azure Activity Log, GCP Audit Logs).
• Regularly test restores: deletion protection reduces risk, backups recover from what gets through.

Key points

• Replacements of stateful resources are the most expensive plan line; make them impossible to miss.
• Layer protections: cloud deletion protection, prevent_destroy, and a CI guard reading terraform show -json.
• prevent_destroy only works while its block exists; cloud-side protection survives a deleted block.