Securing your secrets on Kubernetes

Using Mozilla SOPS to secure your secrets on Kubernetes

Kubernetes has become a de-facto standard for running container workloads. In adidas, it is the central piece of our Open Digital Platform. However, working with Kubernetes has some challenges, like how to deal with sensitive data that your workloads needs for their proper operation. Data like passwords, API keys, tokens, etc.

That data needs to be stored in a secure way as a leak could be very dangerous for us in terms of reputation or revenue. There’s a lot of different solutions for this specific issue. Let’s see how you can store sensitive data safely using Mozilla SOPS without too many modifications.

How a secret looks like in Kubernetes

For newbies to Kubernetes, let’s take a look into how secrets are defined on Kubernetes

A typical secret on Kubernetes

This is the definition for a secret of default type in Kubernetes. As you can see, all the info is stored in a YAML file in a key — value fashion. Values in data section must be encoded in base64. Once created, you can deploy your secret into your namespace using the following command:

kubectl apply -f mysecret.yml

So, now that we know how a secret looks like, there are two questions in terms of security that we need to answer if we want to keep our secrets secure, which are:

• How and where I’m going to store the YAML files securely?
• How to prevent leaks when the secret is deployed to the namespace?

Mozilla SOPS for storing your secrets

Mozilla SOPS is an editor for encrypted files that supports YAML, JSON, ENV, INI and binary files. For encryption, it supports different providers like AWS KMS, GCP KMS, Azure Key Vault, Hashicorp Vault and PGP.
There are too many options for storing our secrets, but for Kubernetes, SOPS it’s a very good one for the following reasons:

• Ease of use: It’s very simple to edit the encrypted files, you just need to decrypt the file, edit it and then encrypt it again. You don’t need to learn a new tool, or interact with a 3rd party service using a REST API which can increase the learning curve of your project.
• Keep your code and you secrets in one place: Encrypting you secrets allows you to commit them to a CVS like git. That simplifies a lot the CI/CD cycles compared to other solutions.
• Keep track of your changes easily: That one is related with the previous one: having your secrets on a CVS allows you to keep track of the changes. Also, it allows you to keep track of who did the change on the secret.
• No additional infrastructure: There are other solutions like Hashicorp Vault that apart from the tool, it also requires you to deploy a storage for them (Consul, DynamoDB…) Those tools add complexity to your architecture which at the end doesn’t add any value for your product.
• No scalability issues: Related again with the previous one, since we don’t need to deploy additional infrastructure for our secrets storage, we don’t need to operate and scale it, which can be a big problem depending on the size of your project.

Having said that, let’s move on how you can encrypt your data using it

Encrypting and decrypting with SOPS

For basic usage, there’s a quick example that you can test using a PGP key on SOPS repository, but the power of SOPS lays on his config file .sops.yaml, where all the magic happens. Lets take a quick look to it:

An example for .sops.yaml file

.sops.yaml contains the rules that will be used for encrypting / decrypting all the files in your repo. On the yaml file provided as example, there’re two different rules for encrypting / decrypting secrets. These rules are evaluated from top to bottom, applying the rule which appears first. Let’s take a closer look into both rules:

• First one, says that all the files matching the provided regular expression will be encrypted / decrypted using AWS KMS. The main advantage of using SOPS in combination with AWS KMS is that you can feed SOPS with multiple keys in different regions, avoiding a single point of failure on the encryption / decryption process
• Second one, states that files will be encrypted using a PGP key. These rule will be used for any file that doesn’t match the previous one (Remember that rules are evaluated from top to bottom)

Once that you’re done with the .sops.yaml file, you can encrypt your secret using the following:

sops -e <filename> -p <your_key_here>

And it will produce something like this:

An encrypted secret resulting of the usage of SOPS.

As you can see, all relevant data on the file is now encrypted. SOPS doesn’t encrypt the keys, so you can still see what has been changed without seeing the actual value.

SOPS also supports key rotation. In case that your keys are leaked, you can remove from .sops.yaml file and reencrypt your secret very easily, for example:

sops -r -i --rm-pgp <your_key_here> filename

Take a look into SOPS documentation if you need more advanced examples about this topic.

So, we already have our secrets stored safely on CVS, but there’s still one pending issue, how can I make sure that your secrets stay safe on the cluster?

Securing your secrets on the Cluster: RBAC

So now that we have our secrets stored safely in our CVS, it’s time to make sure that only people who needs to access them are allowed to.

Usually, when operating a Kubernetes Cluster, we separate projects per namespaces, and we grant all permissions by default on these namespaces. That’s a very bad practice that we must avoid at all costs. Fortunately, Kubernetes provides a very powerful role system that we can use for that purpose.

Let’s start by creating a new role:

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: your_namespace
  name: read-only-role
rules:
- apiGroups: [""]
  resources: ["pods", "configmaps", "services", "pods/log"]
  verbs: ["get", "watch", "list"]

This a very basic role that will allow anyone in your_namespace to execute only a few operations:

• Get / List / Watch, pods, configmaps or services
• Get logs from a pod

And now, bind that role. For that, you’ll need a rolebinding. As the name says it binds a role to a subject. Subjects can be users, groups or serviceAccounts. Use the following example to bind the role to a given user:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-only
  namespace: your_namespace
subjects:
- kind: User
  name: Carlos# Name is case sensitive, so be careful with that!
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: read-only-role
  apiGroup: rbac.authorization.k8s.io

Once applied, even if my credentials are leaked (which I hope not) a malicious attacker will be only able to do the operations described before, reducing the harm done in your infrastructure.

Wrapping up

SOPS can be a very powerful tool for securing your secrets. However, keep in mind that you’ll need to apply more strategies like the one described here related to the cluster to ensure that your secrets remain secret.

I encourage you to play with it and share your strategies on how to handle sensitive information on your workloads in Kubernetes.