Runbooks

Backend Troubleshooting

How to resolve Terraform backend initialization failures with S3 and DynamoDB.

Production

Symptoms

terraform init fails with S3 or DynamoDB errors, such as:

  • Error configuring S3 Backend
  • NoSuchBucket: The specified bucket does not exist
  • AccessDenied errors
  • DynamoDB table not found

Investigation

1. Check AWS Credentials

task aws:whoami

If this fails, your AWS CLI is not configured correctly:

aws configure --profile ontopix-dev

Verify you're using the correct profile:

  • Account ID should match the value in global/variables.tf
  • Region should be accessible

2. Check Backend Resources

task bootstrap:status

This checks for:

  • S3 bucket ontopix-tfstate (in eu-west-1)
  • DynamoDB table ontopix-tflocks (in eu-west-1)

3. Common Issues

S3 bucket not found: The backend hasn't been initialized. Run:

task bootstrap:init

This is idempotent — safe to re-run.

Access denied: Your AWS credentials don't have permission to access the backend resources. Verify:

  • Your IAM user/role has S3 and DynamoDB permissions
  • You're using the correct AWS profile (ontopix-dev)
  • The bucket policy allows your identity

Region mismatch: The backend is in eu-west-1 (Ireland), but your default region might be different. The backend configuration in global/backend.tf specifies the correct region — make sure it hasn't been modified.

Lock table errors: If DynamoDB errors occur during init, it may be a permissions issue or the table was accidentally deleted. Re-run:

task bootstrap:init

Resolution

Re-initialize Backend

task clean          # Remove stale .terraform directory
task bootstrap:init # Ensure backend resources exist
task infra:init     # Re-initialize Terraform
task infra:plan     # Verify everything works

Manual Verification

If automated checks fail, verify directly in AWS:

# Check S3 bucket
aws s3api head-bucket --bucket ontopix-tfstate --profile ontopix-dev --region eu-west-1

# Check DynamoDB table
aws dynamodb describe-table --table-name ontopix-tflocks --profile ontopix-dev --region eu-west-1