Two Applies, One State File

Scenario

Example team: four platform engineers manage a payments platform on AWS with one Terraform configuration. State lives in an S3 bucket so everyone shares it. Engineers run terraform apply from their laptops when their pull request is approved.

What you see in production

On Thursday afternoon Ana adds an ElastiCache cluster and Ben adds an SQS queue. Both applies succeed. On Friday the cache still exists in AWS, but terraform plan wants to create it again, and the next apply fails with "already exists". Nobody touched the console.

The naive code

A remote backend with no locking:

terraform {
  backend "s3" {
    bucket = "acme-terraform-state"
    key    = "payments/terraform.tfstate"
    region = "eu-west-1"
    # no lock: two runs can read and write the same object at the same time
  }
}

Reproduce it

Every run reads the whole state at the start and writes the whole state back at the end. Without a lock, the last writer wins:

import copy

def run_scenario(use_lock):
    backend = {"state": {"serial": 7, "resources": ["db", "vpc"]}, "lock": None}
    cloud = {"db", "vpc"}

    def start(who):
        if use_lock:
            if backend["lock"]:
                print(f"  {who}: Error acquiring the state lock (held by {backend['lock']}) - try again later")
                return None
            backend["lock"] = who
        return {"who": who, "state": copy.deepcopy(backend["state"])}   # plan reads the state

    def finish(run, new_resource):
        cloud.add(new_resource)                                         # the API call succeeds
        run["state"]["resources"] = sorted(run["state"]["resources"] + [new_resource])
        run["state"]["serial"] += 1
        backend["state"] = run["state"]                                 # whole file written back
        if backend["lock"] == run["who"]:
            backend["lock"] = None
        print(f"  {run['who']}: wrote serial {run['state']['serial']} {run['state']['resources']}")

    ana = start("ana")
    ben = start("ben")
    finish(ana, "cache")
    if ben is None:
        ben = start("ben")                                              # retries after ana's apply
    finish(ben, "queue")
    untracked = sorted(cloud - set(backend["state"]["resources"]))
    print(f"  in the cloud: {sorted(cloud)}")
    print(f"  untracked by state: {untracked or 'none'}")

print("Without a lock:")
run_scenario(use_lock=False)
print("With a lock:")
run_scenario(use_lock=True)

Without a lock:
  ana: wrote serial 8 ['cache', 'db', 'vpc']
  ben: wrote serial 8 ['db', 'queue', 'vpc']
  in the cloud: ['cache', 'db', 'queue', 'vpc']
  untracked by state: ['cache']
With a lock:
  ben: Error acquiring the state lock (held by ana) - try again later
  ana: wrote serial 8 ['cache', 'db', 'vpc']
  ben: wrote serial 9 ['cache', 'db', 'queue', 'vpc']
  in the cloud: ['cache', 'db', 'queue', 'vpc']
  untracked by state: none

Root cause

State is a single JSON document. Terraform reads it, plans, applies and writes it back. Object storage happily accepts two writers, so Ben's file, which was based on serial 7, replaced Ana's serial 8. The cache was created in AWS but forgotten by Terraform. Locking turns the read-plan-apply-write cycle into a critical section: a second run fails fast with "Error acquiring the state lock" instead of silently overwriting.

The fix

Use a backend that locks, and make the CI pipeline the only place that applies:

terraform {
  backend "s3" {
    bucket       = "acme-terraform-state"
    key          = "payments/prod/terraform.tfstate"
    region       = "eu-west-1"
    encrypt      = true
    use_lockfile = true   # S3 conditional writes; replaces the older DynamoDB lock table
  }
}

terraform {
  backend "azurerm" {
    resource_group_name  = "rg-terraform-state"
    storage_account_name = "acmetfstate"
    container_name       = "tfstate"
    key                  = "payments/prod.tfstate"
    use_azuread_auth     = true   # locking uses blob leases automatically
  }
}

terraform {
  backend "gcs" {
    bucket = "acme-terraform-state"
    prefix = "payments/prod"   # locking is built in
  }
}

Then harden the bucket: versioning on (you can roll back a bad state file), encryption, public access blocked and access limited to the CI role and a break-glass group. If a crashed run leaves a stale lock, confirm nobody is running and release it with terraform force-unlock <LOCK_ID>.

Trade-offs

• Locking serializes all changes to one state, so a busy monolithic state becomes a queue. Split state by component (see "Blast Radius") rather than skipping the lock.
• -lock=false exists for emergencies; using it routinely brings the lost update back.
• Applying only from CI removes laptop drift and credentials, but needs a break-glass path for incidents.

Detect it in production

• Run terraform plan -detailed-exitcode on a schedule; exit code 2 on an unchanged main branch means state and reality disagree.
• Alert on writes to the state bucket from anything other than the CI role (CloudTrail data events, Azure Storage logs, GCS audit logs).
• Watch for "already exists" errors on create: they usually mean a resource fell out of state.

Key points

• State is read whole and written whole; without a lock the last writer wins.
• Use a locking backend: S3 with use_lockfile, Azure Blob (leases) or GCS (built in).
• Version and restrict the state bucket, and apply from one place: CI.