Runbooks
Drift Investigation
How to detect and resolve infrastructure drift — differences between Terraform state and actual AWS resources.
Production
Symptoms
terraform plan shows changes you didn't make. This means someone (or something) modified AWS resources outside of Terraform.
Investigation
1. Detect Drift
task drift:check
Or for a detailed report:
task drift:report
2. Review the Plan Output
The plan will show the differences between:
- Terraform state — what Terraform thinks exists
- Actual infrastructure — what AWS reports
Common drift causes:
- Manual changes in the AWS console
- Changes made by other tools (CDK, CloudFormation, AWS CLI)
- Auto-scaling or AWS service updates
- Another team member applied changes from a different branch
3. Identify the Source
Check recent activity:
- AWS CloudTrail — who made the change and when
- Git history — was a PR merged that wasn't applied?
- Team communication — did someone make a manual change?
Resolution
You have three options:
Option A: Accept the Drift (Update Terraform)
If the manual change was intentional and should be kept:
- Update the Terraform code to match the current state
- Run
task infra:planto confirm no changes - Commit and PR the updated code
Option B: Revert the Drift (Apply Terraform)
If Terraform's version is correct and the manual change should be undone:
task infra:plan # Verify what will change
CONFIRM=yes task infra:apply # Revert to Terraform's version
Option C: Import New Resources
If new resources were created outside Terraform and should be managed going forward:
terraform import aws_route53_record.new_record ZONE_ID_new-record_TYPE
Then add the corresponding Terraform code and verify with task infra:plan.
Prevention
- Always use Terraform for infrastructure changes (never the AWS console)
- Review plans carefully before applying
- Document any intentional manual changes in a drift report