Recovering from Terraform State Corruption 30 Minutes Before a Board Demo

We provided a cloud infrastructure management platform. Our own infrastructure was managed by Terraform with state stored in an S3 backend with DynamoDB locking. We had a board demo scheduled for 2 PM to show off our new multi-region deployment feature. At 1:30 PM, an engineer ran terraform apply on the production workspace to deploy a last-minute UI fix. Another engineer, not realizing there was an active apply, force-unlocked the state and ran their own apply for a different change.

The concurrent applies resulted in a corrupted Terraform state file. Resources in the state no longer matched reality — some resources were duplicated, others were missing. Running terraform plan showed it wanted to destroy and recreate 47 production resources including our RDS database and ECS services. If anyone had run terraform apply at that point, it would have taken down the entire production environment 30 minutes before the board demo. Our lead SRE immediately recognized the danger and told everyone to stop touching Terraform.

S3 versioning saved us. The lead SRE restored the last known good state file from S3 version history, then used terraform import to reconcile the 3 resources that had actually been created by the partial applies. The entire recovery took 18 minutes. After this incident, we implemented mandatory CI/CD for all Terraform changes (no more local applies), added a Slack bot that announces active Terraform operations, and enabled state file snapshots before every apply.