Blast Radius: Splitting a Monolithic State

Scenario

Example company: all AWS infrastructure (VPCs, EKS, 40 RDS databases, IAM, DNS and every service's queues and buckets) lives in a single Terraform root module with 4,800 resources, applied by one pipeline.

What you see in production

Plans take 22 minutes and regularly hit API rate limits. Every PR waits on the state lock. A typo in a shared locals block planned changes to resources in every team's area, and an engineer adding a queue needed permissions to modify the production VPC.

The naive code

infrastructure/
  main.tf          # network, cluster, databases, IAM, DNS, every service
  variables.tf
  terraform.tfstate (remote): 4,800 resources, one lock

Reproduce it

Planning cost grows with the resources in a state, and a mistake's reach is everything in that state:

REFRESH_SECONDS = 0.25          # average API reads per resource during refresh
CONCURRENCY = 10                # terraform's default -parallelism

stacks_monolith = {"everything": 4800}
stacks_split = {"network": 120, "eks-cluster": 260, "data-orders": 90, "data-billing": 70,
                "iam-baseline": 400, "dns": 300, "svc-payments": 60, "svc-search": 42}
stacks_split.update({f"svc-team-{i:02}": 91 for i in range(1, 39)})   # 38 more service and data stacks

def report(name, stacks, changed):
    size = stacks[changed]
    minutes = size * REFRESH_SECONDS / CONCURRENCY * 10 / 60   # x10: rate limits and retries
    print(f"{name}: change in '{changed}' -> plan refreshes {size} resources, "
          f"about {minutes:.1f} min; worst-case blast radius {size} resources")

report("monolith", stacks_monolith, "everything")
report("split   ", stacks_split, "svc-payments")
print(f"split: {len(stacks_split)} stacks with independent locks, {sum(stacks_split.values())} resources in total")

monolith: change in 'everything' -> plan refreshes 4800 resources, about 20.0 min; worst-case blast radius 4800 resources
split   : change in 'svc-payments' -> plan refreshes 60 resources, about 0.2 min; worst-case blast radius 60 resources
split: 46 stacks with independent locks, 4800 resources in total

Root cause

A state file is a unit of locking, planning, permission and failure. Putting everything in one makes every change pay the cost of everything (refresh time, the lock) and gives every change the reach of everything (a bad expression or provider bug can touch any resource). Permissions can't be narrower than the stack: whoever can apply it can change all of it.

The fix

Split along lifecycle and ownership: things that change together, at the same rate, owned by the same team, belong together.

live/
  prod/
    network/          # VPC, subnets, NAT: changes rarely, platform team
    eks-cluster/      # cluster, node groups: platform team
    data-orders/      # RDS + backups: payments team, deletion-protected
    svc-payments/     # queues, buckets, IAM roles for the service: payments team, changes daily
  staging/
    ... same layout, separate state and credentials
modules/              # versioned building blocks used by every stack

Connect stacks through published outputs, not by reading each other's state:

# network stack publishes
resource "aws_ssm_parameter" "private_subnets" {
  name  = "/platform/prod/network/private_subnet_ids"
  type  = "StringList"
  value = join(",", aws_subnet.private[*].id)
}

# service stack consumes
data "aws_ssm_parameter" "private_subnets" {
  name = "/platform/prod/network/private_subnet_ids"
}
locals {
  subnet_ids = split(",", data.aws_ssm_parameter.private_subnets.value)
}

data "terraform_remote_state" "network" {
  backend = "s3"
  config = {
    bucket = "acme-terraform-state"
    key    = "prod/network/terraform.tfstate"
    region = "eu-west-1"
  }
}
# needs read access to the whole network state (including any secrets in it)
locals {
  subnet_ids = data.terraform_remote_state.network.outputs.private_subnet_ids
}

# Azure: data "azurerm_subnet" "private" { name = "snet-private", virtual_network_name = "vnet-prod", resource_group_name = "rg-network" }
# GCP:   data "google_compute_subnetwork" "private" { name = "private", region = "europe-west1" }

Move resources between stacks without recreating them: add removed { from = ...; lifecycle { destroy = false } } in the old stack and an import block in the new one, then apply both and check that each plan is a no-op apart from the hand-over.

Trade-offs

• More stacks means more pipelines and cross-stack ordering (network before cluster before services). Tools like Terragrunt, Terraform Stacks or a simple pipeline graph manage it.
• Splitting too finely creates chatty dependencies and makes atomic multi-resource changes harder.
• terraform_remote_state is easy, but it couples stacks to each other's state format and requires read access to all of that state; published parameters or data lookups are narrower contracts.

Detect it in production

• Track plan duration and resource count per stack; set a budget (for example, plans under 3 minutes, under ~500 resources).
• Watch for lock contention: the queue time before plans start.
• Review which teams have apply rights on which stacks; one team per stack is the target.

Key points

• A state is a unit of locking, planning time, permissions and failure; size it on purpose.
• Split by lifecycle and ownership; keep environments in separate states and accounts or subscriptions.
• Share values through published outputs (SSM parameters, data lookups) and move resources with removed + import.