Drift: When Reality Changes Behind Terraform

Scenario

Example team: during an incident at 02:00, the on-call engineer opens port 5432 to an office IP on the database security group in the AWS console, and raises the ECS service's desired count from 4 to 12. The incident ends; nobody updates the Terraform code.

What you see in production

Three weeks later a routine pull request changes a tag. The plan also shows the security group rule being removed and the service going back to 4 tasks. The reviewer approves the tag change without reading the rest; the next morning the service is under-provisioned and an office tool can't reach the database.

The naive code

A pipeline that only plans when someone opens a pull request, and reviewers who skim:

on:
  pull_request:
jobs:
  plan:
    steps:
      - run: terraform plan -no-color   # drift appears only inside unrelated PRs

Reproduce it

Drift is the difference between state (the last known reality) and the real world. A refresh reads reality; the plan then compares it with the configuration:

config = {"sg_rules": ["443 from 0.0.0.0/0"], "desired_count": 4, "tags": {"team": "payments"}}
state  = {"sg_rules": ["443 from 0.0.0.0/0"], "desired_count": 4, "tags": {"team": "payments"}}
cloud  = {"sg_rules": ["443 from 0.0.0.0/0", "5432 from 203.0.113.7/32"], "desired_count": 12,
          "tags": {"team": "payments"}}

def diff(a, b):
    return {k: (a[k], b[k]) for k in a if a[k] != b[k]}

def plan(refresh_only=False):
    drift = diff(state, cloud)                 # refresh: what changed outside Terraform
    changes = {} if refresh_only else diff(cloud, config)
    for k, (old, new) in drift.items():
        print(f"  ~ {k} changed outside of Terraform: {old} -> {new}")
    for k, (actual, wanted) in changes.items():
        print(f"  ~ {k} will be updated: {actual} -> {wanted}")
    code = 2 if (drift if refresh_only else changes) else 0
    print(f"  exit code {code}")
    return code

print("terraform plan -refresh-only -detailed-exitcode")
plan(refresh_only=True)
print("terraform plan -detailed-exitcode (would revert the hot-fix)")
plan()

terraform plan -refresh-only -detailed-exitcode
  ~ sg_rules changed outside of Terraform: ['443 from 0.0.0.0/0'] -> ['443 from 0.0.0.0/0', '5432 from 203.0.113.7/32']
  ~ desired_count changed outside of Terraform: 4 -> 12
  exit code 2
terraform plan -detailed-exitcode (would revert the hot-fix)
  ~ sg_rules changed outside of Terraform: ['443 from 0.0.0.0/0'] -> ['443 from 0.0.0.0/0', '5432 from 203.0.113.7/32']
  ~ desired_count changed outside of Terraform: 4 -> 12
  ~ sg_rules will be updated: ['443 from 0.0.0.0/0', '5432 from 203.0.113.7/32'] -> ['443 from 0.0.0.0/0']
  ~ desired_count will be updated: 12 -> 4
  exit code 2

Root cause

Terraform only learns about out-of-band changes when it refreshes during a plan. If plans run only for unrelated pull requests, drift shows up mixed into someone else's diff, where it is easy to approve without noticing, and the "fix" silently reverts an emergency change. Nothing in the workflow forces a decision about the drift itself.

The fix

1. Detect on a schedule, separately from feature work: a nightly terraform plan -refresh-only -detailed-exitcode per stack (or a plan with -detailed-exitcode on the main branch). Exit code 2 opens a ticket or alert.
2. Decide for each drifted attribute:
   • Adopt it: change the code to match reality (for example desired_count = 12, or move it under autoscaling).
   • Revert it: apply the main branch to put things back.
   • Ignore it on purpose: attributes another system owns go in lifecycle { ignore_changes = [...] }.
3. Prevent console writes: read-only console roles in production, with a time-boxed break-glass role whose use requires a follow-up pull request.

resource "aws_ecs_service" "api" {
  # ...
  desired_count = 4
  lifecycle {
    ignore_changes = [desired_count]   # owned by Application Auto Scaling, not Terraform
  }
}

terraform plan -refresh-only -detailed-exitcode   # 0 = no drift, 1 = error, 2 = drift
terraform apply -refresh-only                      # accept reality into state (no infrastructure changes)

Trade-offs

• Nightly drift plans cost API calls and need read credentials on a schedule; large states can hit provider rate limits.
• ignore_changes is a scalpel. Ignoring too much hides real drift.
• apply -refresh-only updates state to match reality but doesn't change your code: commit the matching code change too, or the next plan reverts it.

Detect it in production

• Track a "stacks with drift" count per day; it should trend to zero.
• Cloud audit logs (CloudTrail, Azure Activity Log, GCP Audit Logs) show who changed a managed resource outside the CI role.
• AWS Config, Azure Policy and GCP Security Command Center can flag non-compliant changes in near real time.

Key points

• Drift = reality changed outside Terraform; you see it only when you refresh.
• Run drift detection on a schedule with -detailed-exitcode, not only inside feature PRs.
• For each drift, adopt, revert or deliberately ignore it, and restrict console writes so it stays rare.