Building a dynamic and secure application infrastructure with Traefik and Vault’s PKI secrets engine

In the ever-evolving landscape of modern application deployment and infrastructure management, HashiCorp’s suite of tools has emerged as a powerful solution. Terraform, Consul, Vault, Boundary and Nomad, collectively known as the HashiCorp stack, offer a comprehensive set of capabilities for service discovery, secrets management, orchestration and more.

In this blog, I’ll walk you through the process of setting up parts of the HashiCorp stack and then demonstrate how to enhance it by incorporating Traefik for dynamic application publishing. Finally, we’ll take it a step further by integrating Traefik with Vault’s PKI secrets engine for on-the-fly TLS certificate generation. Above you will find an overview image of what we are going to build.

For the HashiCorp stack we will deploy 3 instances for all of the components: Consul servers, Vault servers, Nomad servers and Nomad clients. And for the Traefik cluster, we will set up 1 controller, 3 proxies and 1 plugin registry.

Part 1: Setting up the HashiCorp stack

1. Consul: Begin by installing and configuring Consul, HashiCorp’s service discovery and orchestration tool. Consul enables dynamic service registration and discovery, providing a single source of truth for your infrastructure’s health and availability. In a more traditional IT landscape we were used to creating and managing static configuration files for the location of our servers / applications. But in a landscape where we work with microservices and/or orchestration tools, Maintaining those static configurations is challenging, thus the need to be able to act on and react to the rapidly changing IT landscape.

2. Vault: Next, deploy Vault to manage secrets and sensitive data. Vault’s robust secrets management capabilities ensure that sensitive information is stored securely and accessed only by authorized users and applications.

3. Nomad: Complete the setup by installing Nomad, HashiCorp’s cluster manager and scheduler. Nomad simplifies the deployment and scaling of applications across a cluster, optimizing resource utilization and ensuring high availability. Nomad supports multiple task drivers, and amongst them the most commonly used include Docker and Podman.

For the installation, configuration and integration of these products, please refer to my HashiTalks 2023 session and have a look at my GitHub repository to help you get started with your own HashiCorp stack.

Part 2: Dynamic application publishing with Traefik Enterprise¹

¹ Note that for this part, one could also use Traefik Proxy, the open-source solution, free of charge.

1. Install Traefik: Integrate Traefik into your infrastructure. Traefik is a modern reverse proxy and load balancer that automatically configures itself based on changes in your infrastructure, allowing you to dynamically publish applications without manual intervention. For this installation, visit my GitHub repository for a guide on how to install this on-premise on your virtual machines.

2. Configure Traefik: Set up Traefik to interact with Consul. This integration enables Traefik to automatically discover new services and route incoming traffic to the appropriate instances, eliminating the need for manual configuration updates. This can be achieved by adding a few lines to your providers block within your Traefik configuration file.

providers:
  consulCatalog:
    exposedByDefault: false
    endpoint:
      address: 127.0.0.1:8501
      scheme: https
      datacenter: velp
      token: <token>

3. Dynamic application publishing: With Traefik in place, you can now deploy new applications or services without worrying about updating routing configurations. Traefik’s automatic discovery ensures that your applications are accessible via user-friendly URLs, making it easier to scale and manage your infrastructure. We will create a Nomad jobspec with the following content.

job "whoami" {
  datacenters = ["velp"]
  type        = "service"

  group "whoami" {
    count = 1

    network {
      port "http" {
        to = 80
      }
    }

    service {
      name = "whoami"
      port = "http"

      tags = [
        "traefik.enable=true",
        "traefik.http.routers.whoami.entrypoints=web",
        "traefik.http.routers.whoami.rule=Host(`whoami.inthepicture.photo`)"
      ]
    }

    task "whoami" {
      env {
        WHOAMI_PORT_NUMBER = "${NOMAD_PORT_http}"
      }
      driver = "docker"

      config {
        image = "traefik/whoami"
        ports = ["http"]
      }

    }
  }
}

Pay attention to the tags specified within the service stanza. These tags define how and if Traefik should handle traffic to this service. In this case we want Traefik to expose this service, only on the entrypoint web (which in this case is tcp/80, plain HTTP) and listens on the FQDN whoami.inthepicture.photo.

When you run this against Nomad, Nomad will then orchestrate the workload to any of its Nomad client’s that meet the requirements of the jobspec file. Within the GUI the job is now in a running state.

If we dig a little deeper into the deployment of this job, we will find the host address and the port that is used to map to the Docker container that is run by Nomad.

We also see the service registration that Nomad is performing to the Consul catalog.

We now move over to Consul and see we have indeed a healthy service registered and we see the tags that correspond with the ones we saw in Nomad.

We then move over to the GUI of the Traefik proxy and see at the bottom of the list that we now dynamically obtained a router from the Consul catalog.

When we click on the router we see the service details. This corresponds to the host address and port we saw earlier within the GUI of Nomad.

So now we can actually test a connection through a web-browser and see if this generates any output.

Part 3: Leveraging Traefik Enterprise and Vault’s PKI secrets engine

1. Integrate Traefik with Vault: Traefik Enterprise offers advanced features, including the support of Vault’s PKI secrets engine as a certificate resolver. To enable this, we include the following lines in our Traefik configuration file.

certificatesResolvers:
  vault-server:
    vault:
      url: https://active.vault.service.inthepicture.photo:8200
      auth:
        token: <vault-token>
      enginePath: pki_int
      role: inthepicture-dot-photo

Here you configure the address of your Vault server, the authentication and where Traefik can request its certificates. In this case this would result in a call to the Vault API endpoint /v1/pki_int/issue/inthepicture-dot-photo.

2. Configure Vault’s PKI secrets engine: Leverage Vault’s PKI (Public Key Infrastructure) secrets engine to generate TLS certificates on the fly. Set up the PKI engine to issue certificates for your applications dynamically, enhancing security by ensuring encrypted communication. This could be done with the following commands.

$ vault secrets enable -path=pki_int pki

$ vault write pki/root/generate/internal \
  common_name="In The Picture Intermediate CA" ttl=1000h

$ vault write pki_int/roles/inthepicture-dot-photo \
    allowed_domains=inthepicture.photo \
    allow_subdomains=true \
    max_ttl=3d

3. Automated TLS certificate generation: With Traefik Enterprise and Vault’s PKI engine working together, Traefik can request TLS certificates from Vault whenever a new application is deployed. This automated process eliminates the need for manual certificate management and enhances security by ensuring that all communication is encrypted with trusted certificates. All we have to do is modify our jobspec to include a new tag to our existing tags and modify our existing entrypoint tag from web to websecure (tcp/443 HTTPS).

tags = [
  "traefik.enable=true",
  "traefik.http.routers.whoami.entrypoints=websecure",
  "traefik.http.routers.whoami.rule=Host(`whoami.inthepicture.photo`)",
  "traefik.http.routers.whoami.tls.certResolver=vault-server"
]

This will instruct Traefik to contact the certResolver we created in step 1 when it is exposed by Traefik. Now when we re-deploy our application, Traefik will pick up the configuration changes as we see here in the GUI.

Notice the changes: We now have a TLS enabled router on the websecure entrypoint. That is the only thing that has changed. When we now visit the same URL in the browser we get a TLS encrypted connection without us having to perform any manual action for the certificate request part.

Step-by-step

To recap what we just achieved, I have marked out the workflow of the connections that are made to each component in the right order.

1. Jobspec file is created and is run against Nomad servers.
2. Nomad servers orchestrate the job to Nomad clients (in our case Docker clients) and the container is started on one or more of the Docker clients.
3. Consul client adds the service to the Consul catalog.
4. The Traefik Controller watches for Consul catalog changes and specific Traefik tags. If the right tags are applied, the service will be added dynamically to Traefik.
5. Within the Traefik configuration, Vault is configured as a certificate resolver, so Traefik requests a TLS certificate from Vault for its defined hostname
6. The Traefik Controller forwards all this information to its ingress proxies and they are ready to handle the requests.
7. Vault issues the certificate with a limited TTL back to the controller, which will be auto renewed at 2/3rd of its lifetime.
8. The ingress proxies will reverse proxy the connection to the container running on the Docker clients, presenting a TLS certificate to the client.

Conclusion

By combining the HashiCorp stack with Traefik and Vault’s PKI secrets engine, we’ve built a robust, dynamic, and secure application infrastructure. This setup empowers you to effortlessly deploy, manage, and secure applications while ensuring encrypted communication through automated TLS certificate generation. The integration of these powerful tools not only simplifies operations but also enhances the overall security posture of your infrastructure. As the landscape of technology continues to evolve, adopting modern solutions like these is crucial for staying ahead and maintaining a competitive edge.