Tips and tricks for AWS SSM Parameter Store with confd

Container startup with confd

SSM parameter store is a service from AWS which allows you to manage configuration parameters for your applications, let that be the environment it runs in, passwords for database access, API tokens or whatever you need. It also gives server side encryption for values that should be kept secret and fine grain access control via IAM.

You can organise your parameters into hierarchies by using forward slash in the parameter names. For example you can choose the root element to reflect your environment (staging, prod etc), the second level for your application and then comes the parameter, it could look something like: /production/client-api/database/host. To learn more about this service, please read the documentation.

For the rest of this post I’m assuming that you have AWS CLI set up on your machine, if not you can find detailed instructions on AWS’s documentation. Also you will need jq and confd for the more interesting sections.

Using the AWS CLI to manage the parameters

Just to have something to work with, we are going to create two parameters.

$ aws ssm put-parameter --name /dev/client-api/database/user --value client --type String
$ aws ssm put-parameter --name /dev/client-api/database/password --value p@ssw0rd --type SecureString

In both cases the output will tell us the version of the parameter. In this case as it was the first value for the parameters, the version will be 1.

{
  "Version": 1
}

Parameters do change over time and to update a value you will need to add the --overwrite option to the command.

$ aws ssm put-parameter --name /dev/client-api/database/password --value p@ssw0rd2 --type SecureString --overwrite
{
    "Version": 2
}

Querying parameters

We can now query the just created parameters. I’m going to use jq to filter the response a bit for the sake of brevity.

$ aws ssm get-parameters-by-path --recursive --path /dev/client-api | jq '.Parameters | map({Name,Value,Version})'
[
  {
    "Name": "/dev/client-api/database/password",
    "Value": "AQICAHjH...",
    "Version": 2
  },
  {
    "Name": "/dev/client-api/database/user",
    "Value": "client",
    "Version": 1
  }
]

The results show the two parameters stored and our password (the SecureString) is actually coming back in encoded form. If you add --with-decryption option to the CLI call the value will come in plain text, decrypted.

Versioning

Versioning comes out of the box and it can be a lifesaver. We are humans and we do make mistakes, by accident someone might update a password that was only stored in SSM. In situations like this you can easily go back and check the previous values for a parameter.

$ aws ssm get-parameter-history --name /dev/client-api/database/password --with-decryption | jq '.Parameters | map({Value,Version})'
[
  {
    "Value": "p@ssw0rd",
    "Version": 1
  },
  {
    "Value": "p@ssw0rd2",
    "Version": 2
  }
]

This is enough basic knowledge for now about the SSM Parameter Store, let’s see how we can use it for our apps.

Using the AWS CLI to set environment variables

For this example we are going to assume that our application requires us to set the database credentials as environment variables before it starts. We can easily do this with the help of the AWS CLI and jq. In the following example we are going to get one parameter and export it as an environment variable. (The -r option will tell jq to return raw output, that means in our case not to put double quotes around a string)

$ export PASSWORD=$(aws ssm get-parameter --name /dev/client-api/database/password --with-decryption | jq -r ".Parameter.Value")
$ echo $PASSWORD
p@ssw0rd2

Although this gets us a value and we can do this for each of our parameters but it doesn’t scale easily, it’s a bit messy as well. Luckily there is a better solution.

Enters confd

confd is a tool that can compile configuration files based on templates from parameters stores like SSM, etcd, consul etc… You can download the binary and read more about it on its github page, but we are going to go through some basics.

Imagine that on the local development environment we are starting to use dotenv files, this helps the developers to quickly change parameters and gives them visibility of the configuration. A dotenv (or .env) file is just a very simple key-value assignment file. We also have a requirement to integrate with Google Maps API so now we need a new parameter for the API key.

PASSWORD=p@ssw0rd2
USER=client
MAPS_API_KEY=42abc

We are going to generate this file with confd which will help the developers to start the project fast without the need to run around and try to figure out which values to use.

To compose files with confd you’ll need to create a resource config and a template file and put them in /etc/confd where confd expects it. The resource config defines the source template, the destination file and which parameter paths should be passed to the template.

The following command creates the directories for confd.

sudo mkdir -p /etc/confd/{conf.d,templates}

In the dotenv.toml resource config we pass in an array of keys that the template is going to use, if we are requesting a parameter in the template that is not included here, we will get an error.

Now that we have these in place we can run confd to generate the dotenv file. We are going to execute it in non-daemon mode -onetime, tell it that the backend we are using is SSM -backend=ssm, we limit the scope of loaded paths to the dev environment only -prefix /dev, and we are explicitly telling which region’s endpoint to use with -node.

$ confd -onetime -backend=ssm --prefix /dev -node "https://ssm.eu-west-1.amazonaws.com"
INFO Backend set to ssm
INFO Starting confd
INFO Backend source(s) set to https://ssm.eu-west-1.amazonaws.com
INFO Target config /target/.env out of sync
INFO Target config /target/.env has been updated

How does this fit in with docker?

With ECS one can pass in environment variables in the task definition. But this exposes some security risk as anyone who can read the task definition will see the environment variables. Instead you can create an entrypoint script that generates the config files for your application with confd. Check the github repository below for an example:

jkrnak/confd-ssm-demo

The idea is that on the container start, the entrypoint script will run, it will start up confd which in turn generates the config files, then the entrypoint passes the execution to the cmd. If there issues in the config templates or perhaps you are referencing non existing keys, confd will fail with an error and your container won’t start.

Export dotenv to environment variables

If you application relies on environment variables you can still generate a dotenv file and export the variables with a simple command before starting the application.

export $(cat .env | xargs)

Using confd for more complex files

Confd is not limited to create only dotenv files, it can handle more complex data types it can loop through keys, you can use filters on the keys or values. In the linked repository above there is an example on creating a yaml file by looping through all keys under /client-api/database.

database:
{{- range gets "/client-api/database/*"}}
  {{ .Key | base }}: {{.Value}}
{{- end}}

Which will result in:

database:
  password: p@ssw0rd
  user: client

Final thoughts

AWS SSM parameter store makes it very easy to store and version parameters and secrets. You can of course set up your own etcd or consul, but once you are in the AWS ecosystem you can save some time if you use their solution.

Setting variables with the AWS CLI is simple, but doesn’t scale too well compared to confd. The CLI also takes up much more space, which might not be a problem if you are running on virtual machines but if you are using containers you probably want to keep them small. Check out the github repo above, it builds two images based on alpine linux, one with the CLI and one with confd. The confd binary is less than 6MB whereas for the AWS CLI you need python and it takes about a 100MB more space.

REPOSITORY        TAG    SIZE
ssm-confd_confd   latest 10.8MB
ssm-confd_awscli  latest 110MB

With confd you can write as complex configuration files as you want or can just stick with a basic dotenv, it’s up to you. It’s a very handy tool at a platform engineer’s toolkit and couple with SSM it makes your life a lot easier to deploy to different environments.