Immutable Infrastructure it is NOT

Engineering and language are funny, because naming is hard. I’m going to pick on “Immutable Infrastructure” a bit. There is a huge difference between infrastructure that you cannot change, and infrastructure that you promise not to change. Outside of hardware I’m not sure I’ve ever seen “Immutable” infrastructure. When people talk about “Immutable Infrastructure” today they are talking about infrastructure that they are trying not to change.

Mutability is a neat property that sits at the core of a bunch of interesting engineering problems and even more philosophical questions about identity. For example, to what degree can something change before it is semantically a different thing? But when we say, “Immutable Infrastructure” we are saying, “infrastructure where individual components cannot be changed — only replaced.” And such a thing just doesn’t exist, and realistically it shouldn’t.

We employ declarative tooling that allows us to describe the desired state of our infrastructure (rather than procedural instructions for building that infrastructure). That tooling does not provide any mechanisms for describing “changes” and supports our thinking of infrastructure as immutable (especially if we tell ourselves to ignore the mountain of procedural tooling, control panels, and live terminal sessions).

And that all sounds great when you say it out loud or as part of some marketing pitch, but its a lie. There is no useful server that does not change. There is no usable and purely declarative infrastructure management toolkit.

Every server will generate logs of all sorts. Every server will have handled a unique lineage of request processing traffic and have a unique internal state shaped by different resource utilization patterns. That isn’t to say that a fleet of machines running the same software where each processes its fair share of the load will evolve into drastically different states. I’m just saying that the servers change. That change is almost never visible to purely declarative tooling.

I believe what we are trying to say is disposable infrastructure. Being able to dispose of and replace a component frees you from the burden of tracking changes and procedures for moving from one state to the next, or some arbitrary previous version. Disposable components can be swapped in and out without much thought to the transition.

When you say disposable, you’re not making any false claims about change. In fact you are highlighting that any mutations on those servers might be lost without careful handling — a much more powerful statement. Further, you’re not blocking yourself from real-world package management best-practices (like auto-updates for security patches). You’re not violating your tenets when you use a mix of declarative and procedural tooling. The most complete stacks will use both.