Services with secrets in DigitalOcean Kubernetes(DOKS)

5 min readDec 18, 2022

If you haven’t read the first part yet, please check it out! I have explained the details of deploying an application into doks there.

Please Note: The secret manager used in this post is SecretHub, which was recently acquired by OnePassword. The post will be updated soon, with instructions to do the same using OnePassword.

Okay, so, this is going to be the second part of this post, in which I already have given a hint that, in the upcoming post, we are going to see how can we deploy applications which need secrets. But more importantly, building the entire pipeline, that will inject those secrets from some kind of secret manager. Let’s start!

Storing some secret credentials!

Shhh!

If you haven’t used a secret manager before, you definitely should. A secret manager, is a piece of software, that helps you to keep credentials, or data that your application may use. You do not want them to be in plain text either in the source code, config, or also want them to have minimal visibility by ensuring some form of access control.

I found secrethub, while looking for a free, easy-to-use, hosted secret management service. I am using it for a couple of years now, it’s amazing, and does exactly what it is supposed to do.

In today’s example, we will be using secrethub as our secret management service. The approach described here can also be applied for any other secret managers like Google Secret Manager, Vault, AWS Secrets Manager etc., only the source code will need tweaking to accommodate a different manager.

Getting Started

First, you should install their cli as we are going to set some secrets to try out this exercise. Considering you have the cli installed, and logged into secrethub, execute the following commands in terminal-

# create a repository to hold our secrets
secrethub repo init my-app/credentials# we can segragate further by defining particular directory
# for particular secrets
secrethub mkdir my-app/credentials/aws# next we create the secrets, each time we do `write` the console
# will wait for us to enter the secret content
secrethub write my-app/credentials/aws/access-key
secrethub write my-app/credentials/aws/access-secret# we can now verify the secrets
secrethub read my-app/credentials/aws/access-key

Using the credentials in application

Now that we have the credentials stored in manager, how do you fetch the credentials, and make them available to your application? There can be multiple approaches you can take, for example-

Having a startup script that fetches the data before the application starts, and make them available as env vars; can be part of your base application setup in docker. The same script can even create a file with the secrets, which your application can read at the runtime.
Having a script that fetches, and then creates kubernetes secret, deployment can then refer it to inject the values as env vars for the application to use. This script can be part of your pipeline.
Utilizing an init-container-pattern in kubernetes. The init-container can run a script, which will keep the secrets ready for your application, either as a file or env vars.

Our goal is not to discuss the pros and cons of these approaches, instead we will be choosing the one, that we think, is most appropriate one to follow- the init-container pattern!

The Approach

We will follow a flow like below:

Creating `ServiceAcccount` using Terraform

If you have followed the first part, then you already know that our entire infrastructure so far is provisioned using terraform. We will be extending that further to make our infrastructure capable of accessing secrets in secrethub.

First, you need to create a service account and respective credential, which will be used by your terraform or IaC scripts. We will use github actions for the IaC pipeline like before, so let’s store the credential in a github secret named SECRETHUB_CREDENTIAL, terraformsecrethub provider can use the credential this way.

A very simple github action will look like below-

Now, using terraform we will provision another service account, with necessary secret repository access, that will be used by our init-container.

The blueprint.tf looks like below-

Pretty simple so far, right?

We are done creating the service account and injecting it into the cluster. The next step should be to retrieve the secrets using the service account.

Fetching Secrets

Let’s say we have a file named app-secrets.yaml in our application .k8s folder. Which will have references to the secrets from secrethub. The file will be referenced inside our application to get access to those values in runtime.

This is a template that secrethub understands.

Our init-container script will be placed in a configmap as well —

How the init-container will get the secrethub credential and access to the file locations defined in the script above, will be clearer when we see the deployment file —

Certain parts in the application container are omitted intentionally, as they are not in the scope of our discussion. But please take a look at the deployment file carefully, as it explains the init-container, necessary volumes and how it renders the secret from template file in correct location.

With the secrets.env now created in the particular location, our application can easily access it. Here’s a sample nodejs code, that you can trigger at application startup —

import fs from 'fs';
import SecretParser from './utils';

export const getSecrets = () => {
   const { SECRET_FILE_DIR } = process.env;
   if (!SECRET_FILE_DIR || !fs.existsSync(SECRET_FILE_DIR)) {
      throw new Error('Secret file location not found');
   }

   return SecretParser.parse(fs.readFileSync(`${SECRET_FILE_DIR}/secrets.env`));
};

That’s it!

We can also use any env loaders available for our application, that can automatically load the values into env. In that case, just ensure to restart the application pods.

Final Notes

I hope, this gives you a working idea of how you can use a secret manager and inject secrets into an application.

For small/growing startups, doks can be a great, decent option to have their infrastructure up and running with less amount of cost but with all the existing benefits that kubernets can bring forth. While kubernetes world is amazing, there are often some important details like secret management which is not straightforward to figure out — secrethub and a pipeline like this, can take you to the right direction in no time. I hope you enjoyed it, and don’t forget to clap if you did!

I am trying to write more these days, and thinking of writing a mix of how-tos, system design as well as engineering management. Let me know in the comments if you are willing me to see write in any other particular areas of tech. Constructive feedbacks are welcome!