Apply Exactly What Was Reviewed

Scenario

Example team: the pipeline runs terraform plan on every pull request and posts it as a comment. On merge, a second job runs terraform apply -auto-approve on the main branch.

What you see in production

PR #812 shows a harmless plan: one tag change on the API load balancer. Between the plan and the merge, PR #809 is merged and applied, and it changed the listener rules the same load balancer uses. The merge of #812 runs a new plan on top, which includes reverting a rule that #809 had just added. Nobody reviewed that plan; it went straight to production.

The naive code

on:
  pull_request:
  push:
    branches: [main]
jobs:
  plan:
    if: github.event_name == 'pull_request'
    steps:
      - run: terraform plan -no-color > plan.txt      # reviewed by humans
  apply:
    if: github.ref == 'refs/heads/main'
    steps:
      - run: terraform apply -auto-approve           # a different, unreviewed plan

Reproduce it

A saved plan records the state it was made against (its lineage and serial). Terraform refuses to apply it if state has moved on since; an unsaved apply just plans again:

state = {"lineage": "a1f3", "serial": 41, "listener_rules": ["/api -> api-v1"]}

def plan(config):
    changes = []
    if state["listener_rules"] != config["listener_rules"]:
        changes.append(f"listener_rules: {state['listener_rules']} -> {config['listener_rules']}")
    if config.get("tag"):
        changes.append(f"tag: none -> {config['tag']}")
    return {"lineage": state["lineage"], "serial": state["serial"], "changes": changes}

def apply_saved(saved):
    if (saved["lineage"], saved["serial"]) != (state["lineage"], state["serial"]):
        print("  Error: Saved plan is stale - state changed since the plan was created; plan again")
        return
    state["serial"] += 1
    print(f"  applied reviewed plan: {saved['changes']}")

pr_812 = {"listener_rules": ["/api -> api-v1"], "tag": "owner=payments"}
reviewed = plan(pr_812)
print("PR #812 plan (reviewed):", reviewed["changes"])

state["listener_rules"] = ["/api -> api-v1", "/v2 -> api-v2"]   # PR #809 merged and applied
state["serial"] += 1

print("merge #812 with 'apply -auto-approve' plans again:")
print("  applied unreviewed changes:", plan(pr_812)["changes"])
print("merge #812 with 'apply tfplan':")
apply_saved(reviewed)

PR #812 plan (reviewed): ['tag: none -> owner=payments']
merge #812 with 'apply -auto-approve' plans again:
  applied unreviewed changes: ["listener_rules: ['/api -> api-v1', '/v2 -> api-v2'] -> ['/api -> api-v1']", 'tag: none -> owner=payments']
merge #812 with 'apply tfplan':
  Error: Saved plan is stale - state changed since the plan was created; plan again

Root cause

terraform apply without a plan file creates a new plan from the current state and code. Anything that changed in between, such as another merge, drift or a provider release, ends up in a plan nobody saw. Also, #812 was planned against a branch that didn't contain #809, so its configuration would undo #809's rule: the branch was out of date with main.

The fix

1. Plan once, save it, review it, apply that file:

terraform plan -out=tfplan -input=false
terraform show -no-color tfplan > plan.txt        # what reviewers read
terraform apply -input=false tfplan               # applies exactly this plan, or fails if state moved on

2. Require branches to be up to date with main before merge (or use a merge queue), so each plan includes everything already applied.
3. Run plan and apply in the same pipeline run with the same commit and provider lock file, keeping the plan file as a short-lived, encrypted artifact (it contains state values).
4. Authenticate CI with short-lived credentials via OIDC instead of long-lived access keys:

permissions:
  id-token: write
  contents: read
steps:
  - uses: aws-actions/configure-aws-credentials@v4
    with:
      role-to-assume: arn:aws:iam::123456789012:role/terraform-payments-prod
      aws-region: eu-west-1

steps:
  - uses: azure/login@v2
    with:
      client-id: ${{ secrets.AZURE_CLIENT_ID }}
      tenant-id: ${{ secrets.AZURE_TENANT_ID }}
      subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
# and set ARM_USE_OIDC=true for the azurerm provider

steps:
  - uses: google-github-actions/auth@v2
    with:
      workload_identity_provider: projects/123/locations/global/workloadIdentityPools/ci/providers/github
      service_account: [email protected]

See GitHub: configuring OpenID Connect in AWS for the IAM role's trust policy.

Trade-offs

• Saved plans go stale whenever another apply lands first; on busy stacks you'll re-plan often. That is the system working; smaller stacks reduce it.
• Plan files contain sensitive values; treat them like state (encrypted, short retention, restricted access).
• Merge queues serialize merges per repository, which slows busy repos but makes "up to date with main" automatic.

Detect it in production

• Log the plan file's checksum in the PR and the apply job; they must match.
• Alert on any apply run without a plan file in production pipelines.
• Track "stale plan" failures: frequent ones mean a stack is too busy or too big.

Key points

• Apply the saved plan file you reviewed; never apply -auto-approve a fresh plan in production.
• A saved plan fails as stale if state changed: re-plan and re-review.
• Keep branches current with main, and give CI short-lived OIDC credentials.