Configuration management: The good, the bad and the ugly
The current landscape when it comes to configuration and key management can be frustrating for developers during the best of times or lead to huge security gaps or cause major outages during the worst of times.
In my time as a software developer I have seen countless solutions to the age old question of: where and how do I store my production API keys?
The current article is not a turn key solution for this problem but my experiences with the current state of the industry and how I am trying to solve it with a few friends.
The Ugly
If we consider the ugly as product API keys and configurations being exposed and accessible to anyone within an organization then buckle in because have I got a story for you.
At my previous work places, I have seen keys being stored directly in GIT repositories, Rancher environment variables and even better, committed directly to .env files that were used by docker images.
Why do I consider this ugly? A few reasons.
First, if anyone can access these keys, then anyone can leak them (whether on purpose or not is irrelevant). One of my previous employers used to store their production API keys straight in bitbucket. No encryption, no specific permission groups, no nothing. On of our QA automation engineers, upon quitting wanted to take his automation scripts with him as a reference. No harm, no foul right? Well wrong. Very very wrong. Unfortunately his scripts contained our production API keys. He ended up posting his scripts on his public GitHub and when we realized, full panic mode was engaged. We spent over 15 straight hours reconfiguring our entire stack.
The Bad
Moving away from storing your API keys in plain text you can use tools like: git-remote-gcrypt, git-secret or git-crypt.
Git-remote-gcrypt allows you to encrypt your whole git repository, so you can’t just pick and choose which files you want to encrypt/decrypt. Also, forget about doing diffs and compares for your files on Github, Gitlab or Bitbucket. You will now have to checkout the repo, un-encrypt it and then compare it to another unencrypted version. At this point, why even bother versioning your codebase if you don’t even know what’s in each commit.
Git-secret and git-crypt are kind of similar. They are both something you can install locally to encrypt files before committing them and decrypting them locally. While this gives you finer grained control over what gets encrypted, now you have to set this up for all your developers, devops and anyone else who may need access to the configuration values.
This definitely solves one problem (the one of public accesses) but creates a whole slew of others. The whole point of committing something to GIT is to be able to keep it source controlled right? Well when you encrypt data it becomes a lot harder to revert and compare versions. You have to check out each previous version, un encrypt it and then hopefully spot the difference. Forget using Github or Bitbucket to compare revisions as that is now completely useless.
Another use case that I have seen is moving your configurations and API keys out of your code base entirely. While the implementation is specific to how you inject the values into your application when it bootstraps (e.g. Kubernetes and configMaps or accessing them via Consul) it all boils down to the same thing. You abstract your API keys from your code (this is good) and then add your configurations to configmaps for Kubernetes or Rancher secrets(this is bad).
Why do I consider this bad? While the keys are not accessible to everyone, keeping track of how configurations change and evolve becomes a nightmare. Imagine you login into Rancher on your development environment and manually add 6 keys to 4 different applications. You code, you code and you code some more, and now you’re ready to deploy to Pre-Prod! Good luck remembering what you added where.
Now let’s say, you are the pinnacle of organization and know exactly what key to add to what application in which environment. That’s awesome! However you have a coworker who isn’t, and accidentally went in and changed a value in your Rancher secret. Now your entire stack is down, and you have no idea why.
So while storing your keys in GIT at least allowed you to see the historical changes of your keys and values, here you are left guessing what went wrong by deciphering pages upon pages of logs.
The Good
So far I have not found a solution that can be considered a silver bullet for configuration management. It always seems to be a win somewhere (i.e. security) and a loss elsewhere (i.e. seeing the changes).
I have in the past used a combination of config maps, docker secrets, Rancher secrets and even Consul (literally the worst).
A few friends and I, are trying to solve this issue with a project called ConfigTree.co
We’re trying to solve this for anyone who has gone through the pain of hunting down the right key value pairs, accidentally changed something and wasn’t able to revert to a previous version or anyone who stays awake at night knowing their stripe secret key is currently live on Github.
If you have any suggestions, questions or experience with the nightmares that come with configuration management I would love to hear from you. Drop a comment below, or send me a tweet at @mlevkov or @configtree. If you enjoyed the read please hit the 👏 button and hopefully it can provide some value to someone else 😄
Also get tips and tricks on how to elevate your Configuration game by dropping your email below 👇
