From env variables to Docker secrets

Courtesy of Pixabay

12 Factor-app

The 3rd item of the 12 factors application manifesto tells us to store the configuration in the environnement.

It also provides example of what this includes:

  • Resource handles to the database, Memcached, and other backing services
  • Credentials to external services such as Amazon S3 or Twitter
  • Per-deploy values such as the canonical hostname for the deploy

We can wonder if this approach is still recommended today, and exempt of risk. In this article we will consider a simple application and see how it needs to be modified to handle such sensitive pieces of information in a safer way.

Applications running in a Docker world

In the last few years, we saw a lot of changes in a way applications are developed and deployed. Mainly because of the huge adoption of the Docker platform, applications are now mainly following the micro-services architecture: they are made of several services (sometimes tons of them) which communicate together.

It’s now very common to define a micro-services application using Docker Compose file format. This format defines the services and the components they use (network, volumes, …). The following is simple example of a Docker Compose file (which default name is docker-compose.yml) used to define a web application composed of:

  • a frontend which make api calls
  • the api used by the frontend
  • the database used by the api

We also specify a volume used to persist the data from the database, and 2 networks to isolate the different parts of the application (front vs back).

version: "3.3"
services:
db:
image: mongo:3.4
network:
- backend
volumes:
— mongo-data:/data/db
deploy:
restart_policy:
condition: on-failure
api:
image: lucj/api:1.0
networks:
- backend
deploy:
restart_policy:
condition: on-failure
web:
image: lucj/web:1.0
networks:
- frontend
- backend
deploy:
restart_policy:
condition: on-failure
volumes:
mongo-data:
networks:
frontend:
backend:

Note: do not try to run this one, it’s just for the example :)

Handling AWS credentials with env variables

We will give a closer look to the api service and we will suppose this one needs some credentials to AWS S3. For instance, it can need those in order to save and retrieve images for a users profile.

The api service is written in Node.js. The code which connects to the Amazon API, using aws-sdk npm module, is something like the following.

// Middleware handling user's profile images
const AWS = require('aws-sdk'),
config = require(‘../config’),
aws_config = config.amazon;
// Configure AWS SDK
AWS.config.update(aws_config.credentials);
// Define S3 bucket
var s3Bucket = new AWS.S3( { params: {Bucket: aws_config.bucket} } )
...
// Upload image object
s3Bucket.putObject(obj, function(err){
if (err) {
log.error(err);
return next(err);
} else {
return next();
}
}

The config module required in the code above defines AWS credentials among some other configuration stuff. As we see here, each element gets its value from an environment variable.

// config.js
module.exports = {
...
"amazon":{
"credentials": {
"accessKeyID": process.env.AWS_ACCESS_KEY_ID,
"secretAccessKey": process.env.AWS_SECRET_ACCESS_KEY,
},
"bucketName": process.env.AWS_BUCKET_NAME
}
};

Then, when running the application through Docker Compose, we specifiy those environment variables through the environment key.

api:
image: lucj/api:1.0
networks:
- backend
deploy:
restart_policy:
condition: on-failure
environment:
— AWS_BUCKET_NAME=BucketName
— AWS_ACCESS_KEY_ID=AccessKeyID
— AWS_SECRET_ACCESS_KEY=SecretAccessKey

This is one way of doing things. But, having those sensitive information in plain text is dangerous. There is also a risk to have them committed to the source control system.

Handling AWS credentials with Docker secrets

There are several ways to handle those pieces of information in a secure way. Using Docker secrets is one of them.

Note: in the current version, secrets can be created in the context of a Swarm which allows an encryption at rest within the Raft logs. You can find some more information about this mechanism in this previous article.

Instead of defining the credentials information in plain text in the environment variables, we create secrets out of them.

$ echo "BucketName"| docker secret create AWS_BUCKET_NAME -
vjp5zh8hwb9dqkvohtyvtifl1
$ echo "AccessKeyID" | docker secret create AWS_ACCESS_KEY_ID -
5txxg3fslf9g5z1o4i19vvmcr
$echo "SecretAccessKey"|docker secret create AWS_SECRET_ACCESS_KEY -
v8g65iwcx1eb6uuwsjzknyi7g

Once created, secrets can be listed.

$ docker secret ls
ID NAME CREATED UPDATED
5x..vm AWS_ACCESS_KEY_ID About a minute ago About a minute ago
v8..7g AWS_SECRET_ACCESS_KEY About a minute ago About a minute ago
vj..l1 AWS_BUCKET_NAME About a minute ago About a minute ago

But their content can not be retrieved. For instance, if we inspect the secret linked to the key AWS_ACCESS_KEY_ID, we only get metadata but not its actual content.

$ docker secret inspect 5txxg3fslf9g5z1o4i19vvmcr
[
{
"ID": "5txxg3fslf9g5z1o4i19vvmcr",
"Version": {
"Index": 12
},
"CreatedAt": "2017–08–13T12:58:50.54021338Z",
"UpdatedAt": "2017–08–13T12:58:50.54021338Z",
"Spec": {
"Name": "AWS_ACCESS_KEY_ID",
"Labels": {}
}
}
]

Once the secrets are created, we can reference them in the Docker Compose file using the top level secrets key.

secrets:
AWS_BUCKET_NAME:
external: true
AWS_ACCESS_KEY_ID:
external: true
AWS_SECRET_ACCESS_KEY:
external: true

Still in the Docker Compose file, we also need to modify the description of the api service so it uses those secrets.

api:
image: lucj/api:2.0
secrets:
— AWS_BUCKET_NAME
— AWS_ACCESS_KEY_ID
— AWS_SECRET_ACCESS_KEY
networks:
— backend
deploy:
restart_policy:
condition: on-failure

If you follow carefully (I would be upset if you dont :) ), you have probably noticed the version of the image has changed to 2.0 in the snippet above. This is due to a change in the api code in order to use the secrets we have created.

When a service needs to have access to a secret, this one is available by default in a temporary filesystem mounted in each container of that service.

The content of a secret is available in /run/secrets/SECRET_NAME in each container of the service

As our application only checks environment variables at this stage, it needs to be updated to take secrets into account. It can be done with a simple module which only purpose is to read a secret from the /run/secrets. This is illustrated in the following code.

// secrets.js
const fs = require("fs"),
util = require("util");
module.exports = {
// Get a secret from its name
get(secret){
try{
// Swarm secret are accessible within tmpfs /run/secrets dir
return fs.readFileSync(util.format(“/run/secrets/%s”, secret), "utf8").trim();
}
catch(e){
return false;
}
}
};

Note: the call to trim() is really important as it removes the trailing “\n”.

We can then modify the configuration file so it uses the get function of the secrets.js module:

...
"amazon":{
"credentials": {
"accessKeyId": secrets.get(“AWS_ACCESS_KEY_ID”) || process.env.AWS_ACCESS_KEY_ID,
"secretAccessKey": secrets.get(“AWS_SECRET_ACCESS_KEY”) || process.env.AWS_SECRET_ACCESS_KEY,
},
"bucket": secrets.get("AWS_BUCKET_NAME") || process.env.AWS_BUCKET_NAME
}

For each key, we first check if it exists as a secret. If it does not, we use the environment variable instead.

Running the application on Kubernetes ?

We saw above that using a secret instead of an environment variable requires the application to be modified a little bit so it can check if the secret exists within the /run/secrets folder. Does that mean we also need to modify the application once again so it can be deployed on Kubernetes and use its approach of secrets management ? Well, not necessarily.

Note: we will not go into all the details of secrets management in Kubernetes but we will show how those secrets can be mounted in the same location as Docker secrets do (that is to say /run/secrets folder).

In order to create secrets in Kubernetes, we first need to encode each piece of information in base64.

$ echo -n "BucketName" | base64
QnVja2V0TmFtZQ==
$ echo -n "AccessKeyID" | base64
QWNjZXNzS2V5SUQ=
$ echo -n "SecretAccessKey" | base64
U2VjcmV0QWNjZXNzS2V5

We can then create a descriptor file containing those values:

// aws-s3-secrets.yml
apiVersion: v1
kind: Secret
metadata:
name: aws-s3
data:
AWS_ACCESS_KEY_ID: QWNjZXNzS2V5SUQ=
AWS_SECRET_ACCESS_KEY: U2VjcmV0QWNjZXNzS2V5
AWS_BUCKET_NAME: QnVja2V0TmFtZQ==

We would then use kubectl to create the secrets from this file.

$ kubectl create -f aws-s3-secrets.yml

Note: Kompose is a great tool to create descriptors files for Kubernetes’ deployments and services out of a Docker Compose file.

The following file is the deployment created for the api service. In this file we’ve also added a part which tells Kubernetes to mount the secrets named aws-s3 into /run/secrets folder. Doing so, no additional change is needed at the application level to use Kubernetes secrets.

// api-deployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
labels:
io.kompose.service: api
name: api
spec:
replicas: 1
template:
metadata:
labels:
io.kompose.service: api
spec:
containers:
— image: lucj/api:2.0
name: api
volumeMounts:
— name: secrets
mountPath: /run/secrets
volumes:
— name: secrets
secret:
secretName: aws-s3
restartPolicy: Always

Note: in a case a service only needs to have access to a subset of the secrets defined (for instance a service which only needs to know the name of the S3 bucket through AWS_BUCKET_NAME), it will see all the other secrets as well. This should be consider when creating the secrets.

Summary

This article is an overview of the way we can use Docker secrets instead of environment variables to handle sensitive information. In most cases, this only require small changes to be done in the application.

Usage of secrets is an important thing that should be considered when deploying an application as it ensures a much higher level of security.