Vault :or How I Learned to Stop Worrying and Love my secrets (Rails + Vault + YAML + Github + SSL)

The Introduction

“Can you keep a secret safe?”

It is never good enough to just nod. My instinct thinks before me. It surprises me with its’ honesty. “Well, it depends upon the secret, no?”.

But I expect my host is not going be very satisfied with my open honesty. So, I answer with a definite “Yes”, reaffirm with a “Don’t Worry” and try to keep him assured with “I am good at keeping secrets”.

“But can you say the same about your application? Is it good at keeping secrets? Or is it more like you?”
Well… Maybe... idk.

With Rails, our naive strategy has always been storing static stuff inside a YAML file. Why? Because it is easy to find, easy to access and easy to change. Now, some people really did not like this. “Secret keys on Github; with code; with devs; Blasphemy!” they said. So, they start keeping secrets inside env files, or passing secrets on to DevOps and automatic provisioners who would later pass them along as .env to the application. The general modern-dev strategy has been to store the secrets inside a .dotfile but again, there is also a considerable amount of community consensus that one should move away from using ENVs to store secrets. Here is a nice article that points out what is wrong with using ENV variables and also gives you some insight to Docker Secrets to start with.

https://diogomonica.com/2017/03/27/why-you-shouldnt-use-env-variables-for-secret-data/

After pondering over strategies that are both secure for the application and adaptable to developers, I stumbled across Vault, a beautiful project that is born out of Hashicorp. It does all the encryption heavy lifting for you while you sit back and talk to it through an HTTP API. That should be more than enough for us to get started.

But before we get going, I would like to tell you about a few things that I personally love about Vault.

1. Independence
   The secrets sit separate from my Github Repository. I don’t store them plainly in my YML file or .env file. I don’t like to Slack my DevOps and ask him to seed a few secret keys in production server for me. I don’t either like depending on my provisioner for secrets related to my application.
2. Access and Access Control
   Vault makes it easy to use Github as an authorization backend. With this, you can limit who gets what level of access to which secrets. With Vault policies, you can give a developer in your team ability to modify only secrets for staging, while still restricting him from playing with your production secrets. And you don’t even need to give him anything to start with. Just ask him to use his Personal Access token from Github. And when he leaves your team, he automatically loses access to the secrets. How cool is that for automation.
3. HTTP API
   You can go really fancy and setup your own User Interface for managing secrets. All what you need is already there; what you want, you can always build.

The Infrastructure

We are going to kick off by setting up a small Vault infrastructure and see if we can get it to play with our Rails app nicely. Here is what we plan to end up with.

1. A Vault Server
2. A consul backend which will serve as a database for Vault
3. An ability to communicate to these two over SSL with LetsEncrypt’s SSL certificate

docker-compose-local.yml

version: '3'
services:
  consul-base:
    image: progrium/consul:latest
    container_name: "consul-base"
    ports:
      - "8400:8400"
      - "8500:8500"
      - "8600:8600"
      - "53:53/udp"
    command: "-server -bootstrap-expect 1 -ui-dir /ui -bind 0.0.0.0 -data-dir=/consul-data"
    volumes:
      - ./consul-data:/consul-data
  consul-agent:
    image: progrium/consul:latest
    expose:
      - "8400"
      - "8500"
      - "8600"
    command: "-server -join consul-base -bind 0.0.0.0"
    depends_on:
      - consul-base
  vault:
    image: "vault"
    depends_on:
      - consul-base
      - consul-agent
    links:
      - "consul-base:consul"
    environment:
      - VAULT_ADDR=http://vault:8200
    ports:
      - "8200:8200"
    volumes:
      - ./config:/config
    command: "vault server -config=/config/vault.hcl -log-level=trace"

We use the official consul and vault images. consul-base is the first consul server we set up. We mount the data-dir for consul as a data volume because we want it to persist across restarts and crashes. We also ask the consul-agent to connect to consul-base container., so that we have a two container cluster.

The Vault configuration is pretty self explanatory. We expose port 8200 for Vault. Remember that we are linking the consul-base container and naming it as consul We will be using this for our vault configuration. Also note that we mount a ./config folder which contains a vault.hcl file.

./config/vault.hcl

backend "consul" {
  address = "consul:8500"
  path = "vault"
  scheme = "http"
}
listener "tcp" {
  address = "vault:8200"
  tls_disable = 1
}
disable_mlock = true

The config file tells vault what its’ address is and what consuls’ address is. Remember we linked the consul-base container as consul This gives us the ability to simply write consul:8500 inside the config file and Docker does the domain name resolution for you. If this is setup correctly, you can now run docker-compose up and you should be able to access consul’s UI at localhost:8500 To see if Vault is working, you can run

docker exec -it vault_container_id vault status

This will probably give you an error along the lines of Vault not being initialized. After Vault is setup for the first time, you will need to run vault init once to get it working. We are going to do that, but we want to operate Vault with SSL support, so we will come back to operating Vault later. Let’s set up some certificates first. It is quite handy to have a server with a valid domain name before you do this.

The free and easiest way to play around with SSL certificates is Certbot. You can generate a very-very valid certificate with just one command without worrying about CAs and signing your certificates and all the drama.Here is the official installation link for certbot. Go ahead and install it first. And then, use the command below and you get a certificate that is ready to go. You could also dockerize this step but I did not. Anyway, run the following command now.

./certbot-auto certonly --rsa-key-size 4096 --standalone -d my.domain.com --email me@gmail.com --agree-tos --text

Certbot is so amazing that it will spin up a webserver automatically for you and generate your LetsEncrypt certificates without you having to bother about it. You will get a bunch of .pem files which we are going to use now for our configuration. Let’s move it inside our config directory and let’s create a config file for consul inside our config directory, since we want Consul to use TLS as well. Since we mount it inside the /consul/config directory inside our container, we use the same path in our config file.

./config/consul.config

{
  "key_file": "/consul/config/privkey1.pem",
  "cert_file": "/consul/config/cert1.pem",
  "ca_file": "/consul/config/fullchain1.pem",
  "ports": {
    "http": 8501,
    "https": 8500
  },
  "verify_incoming": true,
  "verify_outgoing": true,
  "encrypt": "cg8StVXbQJ0gPvMd9o7yrg==",
  "enable_debug": true
}

We will also need to change our config/vault.hcl file so it looks like

backend "consul" {
  address = "my_server_address:8500"
  redirect_addr = "https://vault:8200"
  path = "vault"
  scheme = "https"
  tls_skip_verify = 0
  tls_cert_file= "/config/cert.pem"
  tls_key_file = "/config/privkey.pem"
  tls_ca_file = "/config/fullchain.pem"
}
listener "tcp" {
  address = "0.0.0.0:8200"
  tls_disable = 0
  tls_cert_file = "/config/cert.pem"
  tls_key_file = "/config/privkey.pem"
  cluster_address = "0.0.0.0:8200"
}
disable_mlock = true

You will notice that we used the certificates from certbot and enabled TLS for both Vault and Consul. You will also notice that consul’s address is changed to my_server_address:8500 This is because your SSL certificate is tied to your domain and not your localhost. Other nuances are there to make everything work. Now let’s change our old docker-compose file a bit.

docker-compose.production.yml

version: '3'
services:
  consul-base:
    image: consul:0.7.5
    container_name: "consul-base"
    ports:
      - "8400:8400"
      - "8500:8500"
      - "8600:8600"
      - "8300:8300"
      - "53:53/udp"
    command: "consul agent -server -bootstrap-expect 1 -ui-dir /ui -bind 0.0.0.0 -data-dir /consul/data -config-file /consul/config/consul-config.json -log-level debug -client 0.0.0.0"
    volumes:
      - ./consul-data:/consul/data
      - ./config:/consul/config
consul-agent:
    image: consul:0.7.5
    expose:
      - "8300"
      - "8400"
      - "8500"
      - "8600"
    links:
      - "consul-base:consul"
    command: "consul agent -retry-join consul -bind 0.0.0.0 -data-dir /consul/data -config-file /consul/config/consul-config.json -log-level debug -client 0.0.0.0"
    depends_on:
      - consul-base
    volumes:
      - ./consul-data:/consul/data:rw
      - ./config:/consul/config
vault:
    image: "vault"
    cap_add:
      - IPC_LOCK
    depends_on:
      - consul-base
      - consul-agent
    links:
      - "consul-base:consul"
    environment:
      - VAULT_ADDR="vault:8200"
      - CONSUL_HTTP_SSL=true
    ports:
      - "8200:8200"
    volumes:
      - ./config:/config
    command: "vault server -config=/config/vault.hcl -log-level=trace"

You will notice that I am using the 0.7.5 version of consul image because I found some problems with the latest version; problems that were centred around mounting directories and handling permissions. It is advisable that you use this specific image because it is going to save you a lot of headaches. Now, we have everything in place, we can simply run docker-compose up and watch the magic unfold.

Remember I told you we need to init Vault the first time it starts. Let’s go ahead and do that now with

docker exec -it -e VAULT_ADDR="https://0.0.0.0:8200" vault_container vault init -tls-skip-verify

At this point, vault should start and give you 1 root token and 5 unseal keys. Remember to save these keys securely because you will need them to unlock the vault once it gets sealed. Now, you can check the status of vault with

docker exec -it -e VAULT_ADDR=”https://0.0.0.0:8200" vault_container vault status -tls-skip-verify

This should tell you that the Vault is sealed and you need to unseal it. It is time to use the keys that you received with vault init

docker exec -it -e VAULT_ADDR="https://0.0.0.0:8200" vault_container vault unseal -tls-skip-verify

Do this three times with three different keys and then check vault status again. Things should be rolling now. At this point you can start adding and reading secrets from Vault. Give it a go!

docker exec -it -e VAULT_ADDR="https://0.0.0.0:8200" vault_container vault write foo value=bar -tls-skip-verify
docker exec -it -e VAULT_ADDR="https://0.0.0.0:8200" vault_container vault read foo -tls-skip-verify

Hopefully, your Vault infrastructure up and running; it is now time to connect it with your Rails App.

The Integration

In the last part of this article, we are simply going to hook this infrastructure up with our rails app. You can go ahead and use the Vault Rubygem but I decided to take it a step further and use the gem to store and access secret keys in yml files. For this reason, I created my own gem called Vaml.

The first step was to create policies for the developers in my team. The Vaml gem includes helper methods to help you easily add policies. Here is an excerpt of a rake task to populate policies into vault.

namespace :add do
  desc "Adds vault related policies"
  task :vault_roles do
    general_policy_definition = <<-EOH
      path "sys" {
        capabilities = ["deny"]
      }
      path "secret/*" {
        capabilities = ["create", "read", "update", "delete", "list"]
      }
      path "secret/production" {
        capabilities = ["create", "update"]
      }
    EOH
swat_policy_definition = <<-EOH
      path "*" {
        capabilities = ["create", "read", "update", "delete", "list"]
      }
    EOH
    Vaml.configure(host: ENV['VAULT_ADDR'], token: ENV['VAULT_TOKEN'])
    Vaml::Github.enable_auth(myOrg)
    Vaml.add_policy("vault_general", general_policy_definition)
    Vaml.add_policy("vault_swat", swat_policy_definition)
    Vaml::Github.grant_policy("all_devs", "vault_general")
    Vaml::Github.grant_policy("admins", "vault_swat")
  end
end

When I run run this task with

VAULT_ADDR=https://my_server:8200 VAULT_TOKEN=rootToken rake add:vault_roles

It is going to create two policies and grant them to respective teams within myOrganization. The vault_swat policy applies to admins Github group, and the vault_general policy applies to add_devs group.

Now that all is setup, I create a YML file inside my application that look like

my_app_secrets.yml

development:
 aws:
  access_id: ‘XXX’
staging:
 aws:
  access_id: vault:/secret/staging/aws/access_id
production:
 aws:
  access_id: vault:/secret/production/aws/access_id

The Vaml gem takes in the YAML file, parses it using Psych and returns an object that contains the real value of the stored secrets. In order to save the secrets, the gem also provides a convenience rake task

rake vaml:add_secret /secret/staging/aws/access_id ABC

Since our goal is to access secrets from .yml file, we are goinf to use Vaml again.

Vaml.configure(host: ENV['VAULT_HOST'], token: my_secret_token)
Vaml.from_yaml(File.read "#{Rails.root}/config/my_app_secrets.yml")

The .from_yaml method will raise an error if the keys are not yet stored inside Vault. If they are properly stored, it will return an object that looks like

{
 “development” => {“aws”=>{“access_id”=>”XXX”}},
 “staging” => {“aws”=>{“access_id”=>”ABC”}},
 “production” => {“aws”=>{“access_id”=>”ABC”}}
}

It is up to you to use the value . You can rewrite it into a file but I find it safer to inject it into the Rails app during initialization, doing something like this in application.rb

module MyApp
  def self.vault
    Vaml.configure(host: ENV['VAULT_HOST'], token: ENV['VAULT_TOKEN'])
  end
 class Application < Rails::Application
   ...
config.secrets = MyApp.vault.from_yaml(File.read "#{Rails.root}/config/homify_secrets.yml")[Rails.env]
....
 end

This makes sure that the secrets are only available for us in the Rails process and not written to any YML files. I don’t like using ENV variables for storing VAULT_TOKEN, which is what I was trying to avoid in the first place, however it is a good simple strategy to get things running. Plus, it is easier to use any other strategy to manage just one secret key rather than a bunch of them. If you are using docker, you can probably use Docker Secrets to pass this token along. I would like to hear more recommendations on strategies to pass along the vault token without ENV variable, but for the scope of this article I am going to leave this be.

This should be sufficient for you to get started with Vault. But there is a lot more that you can do with it. You can use the AppRole auth backend and let your CIs communicate with Vault to fill up the secrets. Or you can go really fancy and build your own UI on top of the Vault HTTP API to manage your secrets. The possibilities here could be endless.