Runbooks

Drift Investigation

How to detect and resolve infrastructure drift — differences between Terraform state and actual AWS resources.

Production

Symptoms

terraform plan shows changes you didn't make. This means someone (or something) modified AWS resources outside of Terraform.

Investigation

1. Detect Drift

task drift:check

Or for a detailed report:

task drift:report

2. Review the Plan Output

The plan will show the differences between:

  • Terraform state — what Terraform thinks exists
  • Actual infrastructure — what AWS reports

Common drift causes:

  • Manual changes in the AWS console
  • Changes made by other tools (CDK, CloudFormation, AWS CLI)
  • Auto-scaling or AWS service updates
  • Another team member applied changes from a different branch

3. Identify the Source

Check recent activity:

  • AWS CloudTrail — who made the change and when
  • Git history — was a PR merged that wasn't applied?
  • Team communication — did someone make a manual change?

Resolution

You have three options:

Option A: Accept the Drift (Update Terraform)

If the manual change was intentional and should be kept:

  1. Update the Terraform code to match the current state
  2. Run task infra:plan to confirm no changes
  3. Commit and PR the updated code

Option B: Revert the Drift (Apply Terraform)

If Terraform's version is correct and the manual change should be undone:

task infra:plan    # Verify what will change
CONFIRM=yes task infra:apply   # Revert to Terraform's version

Option C: Import New Resources

If new resources were created outside Terraform and should be managed going forward:

terraform import aws_route53_record.new_record ZONE_ID_new-record_TYPE

Then add the corresponding Terraform code and verify with task infra:plan.

Prevention

  • Always use Terraform for infrastructure changes (never the AWS console)
  • Review plans carefully before applying
  • Document any intentional manual changes in a drift report