When configuration becomes data
What’s actually the difference between configuration and data? Admittedly, a very weird topic to think about, yet I still get pulled back to it regularly.
And, as someone in an ever-growing company, the question seems to become more and more important as we keep adding new developers, users and products in our company.
In DevOps (and Software Engineering, too), we work more and more with Infrastructure as Code (IaC) tools. Ansible, Puppet, Chef, Terraform, Kubernetes… the list of tools that allow you to declaratively define your servers and applications keeps growing.
The file formats are different from tool to tool. Some use JSON, some YAML, some make their own for whatever reasons (hello Terraform!). But overall, they are all configuration files. Most tools, like Helm and Terraform, have templating and some level of scripting built in to make working with larger and more complex configurations easier. And, as our systems and companies grow, those configs become more and more complex, in turn spawning ever more complex a̶b̶o̶m̶i̶n̶a̶t̶i̶o̶n̶s̶ tools to manage the configuration files themselves — like JSONNET.
Configuration management, the overarching name of this entire field, is a fun and exciting topic. As long as you only think about it, academically. Implementing (and more importantly: maintaining) an overlay-based meta-configuration across business domains in practice is, to put it mildly, somewhat less fun.
But this post isn’t about configuration management. Or rather, it’s about the question: At which point does configuration stop being configuration, and simply becomes boring data?
Configuration and data
Well, to be precise about it, configuration is just one specific subset of data. So, for the purposes of this post, let’s make up some arbitrary definitions:
- Data, as we typically use the term, is stored in databases, accessed programmatically, and generally changed frequently during the runtime of a system.
- Configuration, on the other hand, is stored in the above mentioned (human & machine readable) file formats, checked into your git repositories, and a change in configuration typically requires code reviews, deploys, and/or restarts.
We (by “we” I mean everyone who works with DevOps-y things) generally have a good intuitive understanding of the differences between configuration and data.
New purchase on the shop? Obviously goes into the database. New AWS S3 bucket? Put it into the terraform config file!
But, as the company I’m working in grows in size, some lines have started to get blurry.
Growth of… complexity? No, just volume
Each of our developers has their own test VM, which is used to… well, probably to test some stuff. Probably how to screw up our database again — who knows what else developers do. Those VMs are defined & managed with an IaC system, so to add a new VM we simply have to add a new entry in our configuration file.
Sounds pretty convenient, right? Well, it was convenient when we had only about 30 developers. With ten times as much, and the industry typical fluctuation of engineers, this “easy” solution causes more and more work as more people get on- and offboarded each month. And, like all engineers ever, we love automation. So instead of this manual step there’s now an application that allows people to create new VMs in self-service whenever they want.
And the way the application works is quite simple: Whenever someone new requests a VM in a ticket, the app automatically makes a change to our IaC configuration repo, pushes the change, and the CI deploys the new version — thereby creating the new VM.
That’s a totally sensible solution that does exactly what it should. It even re-uses existing structures like the CI pipeline to deploy new infrastructure. Yet, the longer I think about it, the more it sounds completely wrong to me.
With 30 developers, we considered each test VM to be an individual entity where changes deserve to go through the non-trivial process of JIRA Issues, merge requests, code review, and deploys. With 300 developers, we care so little about new test VMs that we didn’t even track how many are created. At least until the CFO showed up with a red face, high pitched voice, and our latest AWS bill — at that point we started tracking the number of VMs again at least.
When configuration becomes data — it hurts
So, the lesson we learned learned here was this: At some point, when there’s more VMs than story points in your next sprint, their management stops being configuration and becomes data. The Kubernetes enthusiasts may say it’s cattle, not pets. Data, which may change quickly and at any time, without JIRA Issues, reviews or more effort than a button click. Data, that no longer belongs into a config file and should instead live in its ancestral home, a database.
By writing an application that automatically updated our Terraform setup via CI, we’ve essentially committed (no pun intended) the greatest sin of software engineering: We used files as a database. A few paragraphs above I made jokes about developers screwing up databases, but the joke’s on me now. I’m starting to question some long-held beliefs. Maybe those finance people were right, and Excel is actually an acceptable database? Maybe SAP really is the pinnacle of software development?
But I must not stray so far from the light. Let’s focus the entire team on how we can improve the VM-provisioning application. We can still save it. If a user requests a new VM, we write an entry into a database. We re-model our application to compare the state in the database with the existing VMs, and create or delete them as required. Sounds a bit similar to what Kubernetes is already doing with etcd, but it could be different enough to warrant its existence. We could even build a web UI for better self serving of your users, which allows them to provision a machine through a single click! Then… oh damn it, we’re back at the place we were before IaC.
Boundaries are very important, in both SE and SM
Maybe, just maybe, there’s not a single approach to managing applications and systems that is actually “correct” at all scales.
Maybe, as much as I hate to admit it, both IaC and colourful clicky-button-do-thing tools have their place in modern tech companies.
Maybe there’s a breakpoint where configuration becomes data, after which point our lack of tight-gripped control over configuration matters less than easy self-service UX for users.
And maybe we simply have to figure out where the breakpoints are, so that we can decide how the best solution for today and tomorrow should be implemented.
As a developer at heart, I love clear requirements almost as much as automation. Consequently, I’ve spent far too much time, trying to define precise rules that would allow me to algorithmically decide whether an entity should be treated as configuration or data. All that effort, only to end up with a muddy collection of notes that can barely be called guidelines. Here they are:
- If there are more than 14 (now referred to as a “DevOps’ dozen”) practically equal entities in the same environment, it’s probably data
- If you can tell your CTO “honestly, I have no idea how many entities there are, probably a lot” without pissing them off, it’s probably data
- If entities are created/deleted once a month or more frequently on average, it’s probably data
- If only a single person (or a single team, in large companies) would notice or care if an entity were to suddenly disappear for a bit, it’s probably data
- If a change in an entity can, even in the worst case, only create as much monetary damage for your company than your conference budget, it’s probably data
- If an entity isn’t just a sub-component of an overarching architecture, but is instead actually a defining a core part of your architecture, it’s probably not data
Writing out those rules made me realise how many things we currently store in configuration files that don’t belong there. We use Vault everywhere for secrets, and it’s well integrated in our IaC workflows. Maybe using Vault, or something similar that’s not focused on secrets, for storing things that have been promoted from configuration to data could be a good idea 🤔
