A plan on a Tuesday morning showed sixty-two changes across four workspaces. The pull request under review altered one tag on one S3 bucket. Nothing in the repository explained the other sixty-one lines, and the diff was full of resources nobody on the team had opened in months.
The commit was innocent. What had changed was the AWS provider. Our root modules declared >= 5.0, the lockfile had been deleted during a migration to a new CI runner image months earlier, and terraform init in a fresh container had happily resolved the newest release. Between the version our state was written by and the version now planning, the provider had changed how it treats an argument we had never written: bucket-level defaults that used to be absent in state now materialised as explicit values, and a handful of aws_security_group_rule descriptions normalised. Terraform did not care that we had not asked for any of it. Our configuration was the same, our state was the same, and the thing translating between them was different.
Most of the sixty-two changes were harmless. Three were not, including one that would have replaced a launch template and rolled every node in an autoscaling group in the middle of the working day. Nobody would have caught that by reading a plan summary line that says "62 to change".
Two rules came out of it. First, .terraform.lock.hcl is committed, non-negotiable, and CI runs terraform init -lockfile=readonly, so a provider upgrade is a pull request with a plan attached rather than a side effect of a cache miss. Second, every provider block pins a pessimistic constraint, ~> 5.42, so the range of surprise is bounded even if the lock is somehow bypassed.
We also stopped treating a large plan as noise to be scrolled past. The pipeline now fails the plan stage if the change count exceeds a threshold for the workspace, and the only way past it is a human writing why in the pull request.
Your infrastructure is defined by your code plus the exact provider binary that interprets it. Pin both, or you are running a version you never chose against a system you cannot afford to surprise.
– Sergey Shinder
Top comments (0)