According to my plan, this topic should come later in this series, because normally after you talk about infrastructure as code, you talk about configuration management. And since secrets management is a topic related to many other things (like how to use it inside your configuration management, how to use it in your compute resources like Kubernetes), it’s better to introduce this topic after discussing configuration management and Kubernetes.
But because recently I did quite some work on this topic, I’d like to share it now to make hay while the sun shines as a summary of my recent thoughts and work. So forgive me for being illogical this time.
In the field of software development and information technology, no matter what you do, chances are, you can’t do it without secrets.
When you are developing an app, the app itself might need access to a database, so it needs the password. Or maybe the app talks to another API that requires authentication, so you need a token, which is a secret.
You can’t put the secret in your version control system either because that’s dangerous.
When you created the database you were going to use for that app, you need to generate a secret and save it somewhere safe.
When you launch your virtual machine, you might need to provide an initial password too.
In your CI system, you might want to write back something to your version control system (for example, build status, tags, etc.) so you gotta create a user for your CI and save your password somewhere safe so that your CI system can read it.
In your CD system, you might want to SSH into a machine using a private key to do deployment; or maybe you don’t use virtual machines, but instead, you have containers running in a Kubernetes cluster, in which case, you still need to manage the access from your CD system to your Kubernetes cluster.
Secrets are literally involved in every single stage of the software development cycle, and that is why you need to manage them properly.
How Do I Manage Secrets Before without Any Secret Manager
Let’s have a look at how we manage secrets without a secret manager, which is quite common, I would say, in many projects:
1 Storing in Git
You know you should do this.
Because projects can be cloned onto multiple machines, forked into new projects, distributed to customers, made public so on and so forth. Each time it’s duplicated on Git, the entire history of that project is also duplicated. Why storing secrets in public repositories is bad is obvious.
But maybe, in the beginning, the project chose “convenience” over “security” and it was set up like this and you hard-code your secret in your Git repo. maybe someone thought it was a private repo and it’s less dangerous to do so.
2 Storing in Locally
Or maybe you don’t feel safe to store secrets in Git anyway; after all, everyone is talking about how you should not save your secrets in your Git repo.
So you save your secrets in a file locally and add the file into your
.gitignore file so that it isn’t managed by the version control system.
This isn’t the worst bad idea if you are working on the project alone, to be honest; but when the number of people who are working on the same thing grows, it’s hard to scale.
Sharing your secret file via slack? What if, over time, one’s file content got changed, but the other is not synchronized?
3 Storing in Git, Encrypted
So you are looking for a way to encrypt the file which contains secret first, then pushing it to your repo.
Well, this could work, and Ansible vault uses this method to manage secrets used in your configuration management code.
For non-Ansible usage, there are also tools designed to help you do so securely and automatically, for example, git-crypt, which enables “transparent encryption and decryption of files in a git repository.”
Files that you choose to protect are encrypted when committed and decrypted when checked out. git-crypt lets you freely share a repository containing a mix of public and private content. git-crypt gracefully degrades, so developers without the secret key can still clone and commit to a repository with encrypted files. This lets you store your secret material (such as keys or passwords) in the same repository as your code, without requiring you to lock down your entire repository.”
This is a great tool for automating the encryption/decryption process that otherwise would have been done manually if you want to encrypt a file, then store it in Git.
There are downsides, though:
- You must create and manage GPG keys which is overhead because it’s yet another thing you need to worry about besides what you already have got, like your SSH private key, your AWS access key, and secret.
- If the same secret must be used in different repos, you must copy-paste them to another repo and manually maintain multiple copies of data in different places, which will eventually cause inconsistency.
- If the system needs the secret but doesn’t need Git, it’s also overhead. Like your CD system, which might need access to AWS, for example, to run some commands but doesn't really need anything from Git.
4 Secrets in Kubernetes
In case you are mainly using Kubernetes, you need to manage K8s secrets, which are defined in YAML. But for the same reason previously mentioned, it can be tricky to manage or even unmanageable:
When you create a secret, you write a YAML file. But you can’t save it in your Git repo. This is the same as scenario 2 above because you can only save it locally, which has the same downside: it doesn’t scale, and you risk data inconsistency.
Or maybe you decide not to use YAML at all; only editing/creating secrets using kubectl, and use your K8s as the secret management tool.
In this way, there will not be inconsistency, data is only stored in one place, and your app can use those secrets directly. But the problem is, if other systems (like your CI/CD, configuration management/infrastructure code which don’t need K8s to run), it’s not easy to read those secrets.
Secret managers are created to solve these issues.
What a secret manager does is actually quite simple: it stores your secrets somewhere safely, and you can interact with your secrets in multiple ways so that in each stage of your software development cycle, you can refer to it.
There are multiple choices on the market, but here we will talk about two options, one is HashiCorp Vault, because it’s “vendor-neutral” if you are using public cloud providers; and the other is AWS Secrets Manager.
Vault is HashiCorp’s secret manager.
At the moment, you must install it yourself; the cloud version is only in beta status. While this can be a downside for some, it can be a great benefit for others who require the data to be stored at places they want.
If you choose to set it up and maintain it yourself, it requires a little bit of work to make it production-ready, because you need to work out the backend storage: where to store those secrets, how backup is managed, etc. For installing vault to be production-ready, check out my previous article here:
Production Grade Password Manager for Kubernetes and AWS — An Intro to Hashicorp Vault
A basic introduction to k8s secrets and hashicorp vault: what it is, how it’s implemented, and why it’s good.
Managing the access control requires some detailed configuration in HCL configuration language. But once it’s finely tuned, you don’t have to worry about it anymore.
Vault can be integrated with many things, like, you can use Kubernetes/AWS to authenticate with Vault.
Vault supports not only key-value secrets but also other types, like AWS access key, DB password, SSH, etc. It’s probably the most comprehensive secret manager out there but as mentioned, there needs to be some work done to fully utilize it.
AWS Secrets Manager
AWS Secrets Manager is AWS’s answer to the HashiCorp vault. It’s a cloud service that requires no provision to set up, but you can’t choose the backend —where to store these secrets (it’s in AWS, can’t be your datacenter).
You don’t have to worry bout this though, because AWS Secrets Manager uses AWS KMS for encryption.
The biggest benefit of AWS Secrets Manager is, the access control is done via IAM, which if you are already using AWS, you should be already quite familiar with.
AWS Secrets Manager probably is less fancy compared to HashiCorp vault which has way more features, but it does what it is supposed to do pretty well.
Another big benefit of AWS Secrets Manager is, it integrates well with other AWS services. For example, if you are using HashiCorp Vault and you want to be notified when a secret is created/updated/deleted, there isn’t a way to do so because it doesn’t support webhook; you must poll to get the update. With AWS Secrets Manager, it’s the same (that it doesn’t support webhooks to notify you), but because it’s an AWS service, you can capture API call event with CloudWatch rule, then target it to some Lambda to notify you of the change. This “feature” (not exactly a feature of Secrets Manager, but a feature of AWS solution as a whole) can be quite handy, which we will discuss in-depth in the next chapter.
Best Practice: Single Source of Truth
Secret managers are usually much easier to use, than, for example, manually editing a YAML file in Kubernetes. They have beautiful UI as well because humans are meant to interact with them.
But this isn’t the main point. The biggest advantage is, they support different ways to interact with them (human, UI, API, CLI, Kubernetes, etc) so that you can use it as the single source of truth.
Simply using a secret manager doesn’t really solve your chaotic problems.
For example, in the context of Kubernetes, if you store secrets into the secret manager, but every time you need to create a K8s secret, you manually login to the UI, get the values, paste into your local YAML, then apply, there still can be problems, because you are essentially still manually creating those secrets with local files which can be error-prone.
What can improve the secret management situation is, to use the secret manager as the single source of truth.
That is, in the same Kubernetes example above, you don’t manual-get-secret-then-create-apply YAML, but instead, the K8s secret should be created automatically using the secret manager as a source. When the source is changed, the secret in K8s should also reflect the change, without human intervention.
In the next chapter, we are going to have a detailed look at how to automate this process in three different ways and choose the best.