Terraform in Production · Capstone Case Studies

Capstone: A Production Terraform Setup for a Growing Company

Twelve engineers, three environments and two clouds on the horizon. Design how they write, review and apply Terraform safely.

Scenario

Example company: a 30-person fintech start-up runs everything on AWS in one account, built by hand in the console plus a few Terraform files on laptops. They're preparing for a SOC 2 audit, hiring a second platform engineer, and a large customer requires a deployment in Azure within a year.

Today: one VPC, an EKS cluster, 6 RDS Postgres databases, ~40 S3 buckets, SQS queues and IAM users with access keys. Engineers deploy apps through GitHub Actions.

Requirements

Separate dev, staging and prod with no shared credentials.
Every infrastructure change is reviewed, auditable and applied from CI.
No secrets in state, Git or CI logs.
A mistake in one service's infrastructure can't delete a database or the network.
Existing console-built resources are brought under Terraform without downtime.

Deliverables

Account/environment and state layout.
The CI workflow from pull request to apply, including authentication.
How you prevent and catch destructive changes.
How you adopt the existing resources.
What changes for Azure, and what stays the same.

Hints

One state per environment per component; accounts are the strongest boundary on AWS.
Who can read state matters as much as who can apply.
import blocks plus -generate-config-out turn adoption into reviewed code.

Draft each deliverable before revealing the model answer and rubric.

Further reading: S3 backend · Import · GitHub OIDC in AWS

Deliverables

Design the account/environment and state layout.

Model answer

AWS Organizations with separate accounts: management, security/log-archive, shared-services (state bucket, CI roles, ECR), dev, staging, prod. Service control policies deny deleting CloudTrail and leaving the organization.
State: one S3 bucket in shared-services (versioned, KMS-encrypted, public access blocked) with use_lockfile = true; key per environment and component: prod/network, prod/eks, prod/data-orders, prod/svc-payments...
Code: modules/ (versioned building blocks) and live/<env>/<component>/ root modules; the same modules in every environment, with different variables.

Rubric

Full marks: separate accounts (not just workspaces) per environment, state split by component, a locked and versioned encrypted backend, shared versioned modules. Partial: environments only as workspaces in one account, or one state per environment.

Describe the CI workflow from pull request to apply.

Model answer

PR opened → fmt -check, validate, tflint/policy checks, terraform test for modules.
plan -out=tfplan per affected stack, using a read-only plan role assumed via GitHub OIDC; post terraform show output plus a destructive-actions summary as a PR comment.
Approval from the stack's code owners; branch must be up to date with main (or a merge queue).
On merge, the apply job assumes an apply role (only trusted from the main branch and a protected GitHub environment with required reviewers for prod) and runs apply tfplan; a stale plan fails and re-plans.
Nightly plan -refresh-only -detailed-exitcode per stack for drift; exit code 2 opens an issue.

Rubric

Full marks: OIDC with separate plan/apply roles, saved plan applied, protected environment for prod, drift detection. Deduct for long-lived access keys, apply -auto-approve of a fresh plan, or laptop applies.

How do you prevent and catch destructive changes?

Model answer

Data in its own stacks (data-orders) owned by the payments team; service stacks can't touch it.
deletion_protection = true on RDS, prevent_destroy on stateful resources, S3 buckets with versioning and Object Lock for critical data.
CI guard reading terraform show -json that blocks delete on protected types without an allow-destroy label and a second approver.
Backups (AWS Backup with a vault in the log-archive account) and quarterly restore tests.

Rubric

Full marks: at least three independent layers (cloud protection, Terraform lifecycle, automated plan guard) plus tested backups.

How do you adopt the existing console-built resources?

Model answer

Per component: write import blocks for the existing IDs, run terraform plan -generate-config-out=generated.tf, clean the generated code into modules, and iterate until the plan is 0 to add, 0 to change, 0 to destroy. Apply (which only records state), then remove console write access for that component. Rotate IAM user access keys to roles as part of the move. Secrets: switch RDS to manage_master_user_password before importing so passwords never enter state.

Rubric

Full marks: import blocks with generated config, a no-op plan before apply, console access removed afterwards, secrets handled.

What changes for Azure, and what stays the same?

Model answer

Same: workflow, review rules, module versioning, drift detection, plan guard (with Azure types added), state-per-component layout.

Different: subscriptions (under management groups) instead of accounts; an azurerm backend in a dedicated storage account (blob lease locking, Entra ID auth); GitHub OIDC to an Entra ID app registration with federated credentials (ARM_USE_OIDC=true); azurerm_management_lock for delete protection; Key Vault/Entra ID authentication for secrets. Modules are per cloud: share the interface and conventions, not the implementation.

Rubric

Full marks: keeps the process identical and maps each AWS mechanism to its Azure equivalent; doesn't try to write one module that abstracts both clouds.

Open in the app to mark lessons complete, take notes and track your progress.