The Gift of Idempotence
What is idempotence in the context of system administration and why do we need it? First a quick definition.
Idempotence is the idea of non-change when change is unecessary.
First lets start with a few task that are already idempotent by default on a given system. Installing a package and starting a service. These are the bread and butter of any system admin. The first run will change the state of the system. Running the same commands after that will respond with a status message of service is running or package is installed, assuming everything goes accordingly.
When a command is known to be idempotent on a system it gains an additional property, one of status indicator.
By contrast, lets look at the service restart command. Every time you run it the state of the system changes. It is not idempotent and cannot be used as a status indicator.
Using programming logic one can take advantage of these properties and create the basis of a configuration management system. There are many but I will focus on ansible.
What does ansible do? In simpliest terms it wraps every command on the system in modules that check for idempotence and returns whether or not a command changed the system. At the end of the run you get a summarized status of every command.
ok=0 changed=1 unreachable=0 failed=0
The goal is, by the second run, ok should be the only field that is not zero.
ok=1 changed=0 unreachable=0 failed=0
It is in this way that you know your system is in a sane state.
While this essay is about idempotence in particular, I wanted to point out another advantage of having your configuration in a system like this. It is code. Code can be recorded, changed, and versioned in the same way as any other project. If you ignore the unreliable history feature of a linux system, whenever you enter a command it is essentially lost forever, only its effects remain, which are vulnerable to being overwritten by whatever command comes next.
For many years this wild west process of system administration was dealt with by creating installation guides. These guides were prone to human error and the human was required to be especially sensitive to the current state of the system. Freeing our minds from such a task sets us up to spend our time on other more important matters.
In the end a configuration management system is really just a clever accounting system. This accounting relies on the idempotence property. Creating a command without this property creates accounting errors. Accounting errors have the power to crumble multi-national corporations and even entire empires! So stay safe, stay idempotent!