Someone Opened Port 22 to the World Six Months Ago, and Terraform Has No Idea

A security audit flags a security group with 0.0.0.0/0 open on port 22. Nobody on the team remembers adding it, and when you check the Terraform code for that security group, it says the rule should only allow SSH from the office IP range. The infrastructure and the code describing the infrastructure have quietly disagreed with each other for six months, because someone opened the port from the AWS console during an incident, fixed their problem, and never closed it again — and nobody's terraform plan ran on a schedule to notice.

This is infrastructure drift, and it's one of the most common ways "we manage everything as code" turns out to be aspirational rather than true. Terraform only knows about changes it makes itself. Anything clicked in the console, run via a one-off CLI command, or changed by another tool exists in a blind spot until someone happens to run a plan against it.

The Fix #

Turn on AWS Config across every region and account you actually use, aggregated centrally if you're running AWS Organizations. A Config recorder that's only enabled in one region misses every drift event everywhere else.

Layer managed rules with rules specific to your org's actual policies. Start with the obvious ones — restricted-ssh, s3-bucket-public-read-prohibited, iam-user-no-policies-check — then add custom Config rules, backed by Lambda, for anything specific to how your team actually works.

Wire auto-remediation where it's safe to act automatically, using SSM Automation documents attached to Config rules. AWS Config ships a large library of prebuilt AWSConfigRemediation-* automation documents — check what's already available for your specific rule before writing a custom one from scratch.

Run scheduled Terraform drift detection separately from Config. These solve different problems: Config catches real-time resource changes as they happen; a scheduled terraform plan catches drift between what your code says and current reality, on your own timeline:

terraform plan -detailed-exitcode -out=tfplan
EXIT_CODE=$?
if [ $EXIT_CODE -eq 2 ]; then
  echo "Drift detected — posting to Slack"
  # notify pipeline, don't auto-apply blindly
elif [ $EXIT_CODE -eq 1 ]; then
  echo "Plan failed"
  exit 1
fi

Run this on a schedule — nightly, or at minimum weekly — not only when someone happens to push code.

Consider blocking the riskiest manual changes outright with SCPs, rather than only detecting them after the fact. Denying non-pipeline principals from modifying security groups directly, for instance, stops the drift from happening at all instead of catching it six months later.

The Gotchas #

• Auto-remediation without an exception mechanism can revert a legitimate emergency fix. Someone opens a port by hand during a real incident to unblock something urgent — if your remediation Lambda reverts it thirty seconds later with no way to flag "this was intentional, hold off," you've just re-broken the incident fix mid-incident.
• AWS Config has real, scaling costs — per configuration item recorded and per rule evaluation. An account with a lot of resource churn, like frequent scaling or ephemeral resources, can rack up a bigger Config bill than people expect; scope recording to what you actually need to track.
• terraform plan -detailed-exitcode only catches drift when it actually runs. A pipeline that only plans on push, with no scheduled job, leaves weeks of drift invisible between deploys — schedule it independently of your deploy triggers.
• Not all "drift" is real drift. AWS-managed fields — AMI auto-updates, default security group rule reordering, auto-assigned tags — show up as diffs that aren't actual problems. Tune ignore_changes lifecycle blocks carefully for specific known-noisy attributes; don't reach for a blanket ignore that hides genuine drift along with the noise.
• Detecting drift isn't the same as understanding why it happened. A recurring drift on the same resource usually means a process problem — someone keeps needing console access for something IaC doesn't support yet — not just a compliance checkbox to tick.

TL;DR #

• AWS Config catches real-time manual changes; scheduled terraform plan -detailed-exitcode catches drift between code and reality — you need both, they're not redundant.
• Auto-remediation needs an exception path, or it will eventually revert someone's legitimate emergency fix mid-incident.
• Not every diff is real drift — tune ignore_changes for AWS-managed noise instead of either ignoring everything or chasing false positives forever.