What is configuration drift, and how does immutable infrastructure prevent it?

#Two servers that started identical

Configuration drift is what happens when machines that began as exact copies of each other slowly stop being copies. You provision ten application servers from the same base image, they pass the same smoke tests, they behave identically for the first month — and then, six months later, server 7 is running a different minor version of the runtime because someone SSHed in during an incident and patched it by hand, server 3 has an extra cron? job nobody remembers adding, and server 9 has a stale environment variable pointing at a decommissioned database replica. No single change was a big deal on its own. The accumulation is what kills you: eventually a deploy that works fine on nine servers throws an unexplained error on the tenth, and the only way to find out why is to diff two machines that were supposed to be identical and never got audited.

Drift almost always enters through the side door of "quick fixes." A production incident needs an immediate patch, someone logs in and edits a config file directly instead of going through the deployment pipeline, and the fix works — but it now exists only on that one instance, invisible to version control, invisible to the next person who provisions a server the "normal" way. Autoscaling makes this worse, not better: a new instance that spins up during a traffic spike is built from whatever the base image or last full provisioning run captured, which is often already stale relative to the hand-patched instances sitting next to it in the same load-balancer pool. Now your fleet is not just drifted from its original spec, it is internally inconsistent with itself, and which version of "normal" a given request hits depends on which instance the load balancer happened to route it to.

Configuration management tools (Ansible, Chef, Puppet, Salt) were the first generation's answer, and they genuinely help — but only if you run them continuously and treat manual changes as violations to be corrected, not just documented. A tool that reconciles state once a day still leaves a 24-hour window where a manual change is live and undetected, and if engineers know the tool will "fix it eventually," the temptation to skip the pipeline for a quick SSH fix never goes away. The deeper problem is that these tools are typically applied to a machine that already has history — packages that were installed and never removed, files left over from three deployments ago, a /tmp directory nobody cleans. Convergence gets you closer to the desired state; it does not erase what accumulated before the tool started managing that box.

#Replace, don't patch

Immutable infrastructure solves drift by removing the operation that causes it: nobody is ever allowed to change a running instance. Instead, every change — a dependency bump, a config value, a security patch — goes into the source of truth (a Dockerfile, a Packer template, an AMI build pipeline), a brand-new image is built and tested, and the fleet is replaced with instances booted from that new image. The old instances are terminated, not patched. If the new image has a bug, you do not debug it in place; you roll the fleet back to the previous image, which you still have, byte-for-byte identical to what was running an hour ago.

This forces a few disciplines that are worth calling out because they are what people actually get asked about in interviews. First, instances must be stateless: anything that needs to survive an instance's death — session data, uploaded files, application state — has to live outside the instance, in a database, object store, or cache, because the instance itself is disposable by design. Second, configuration has to be injected at boot (environment variables, a secrets manager, a metadata service) rather than baked in as a fixed value or edited after the fact, so the same image can serve dev, staging, and prod without being different images. Third, "fixing" something in production is now, definitionally, a full round trip through the build pipeline — which feels slower in the moment and is exactly the property that makes the fleet trustworthy: there is no longer a way for a fix to exist that isn't in version control.