Using Puppet Deferred functions to provision secrets on your infrastructure

A small intro

At Hootsuite in our Measure portfolio, we use Puppet 5 to provision a handful of components in our infrastructure and some of these components also contain secret keys or tokens that must not be seen by other nodes or be stored on disk like the other static configurations or templates that we have. These keys are fetched from a secret store and are decrypted only at runtime on the targeted component versus the classic process of compiling a catalog on the server and sending it to the agent.

There were many solutions for doing this, but the simplest way was to use the built-in configuration data lookup system Hiera. This solution had multiple advantages like natively integrating with Puppet but also had some disadvantages, not having a granular ACL in place and the configuration for integrating Hiera with Vault provided by the 3rd party was not very pleasant to work with.

Another potential solution was to use consul-template or vault-ctrl-tool but we decided we’re going to go another route and try out Puppet’s Deferred functions. One of the reasons for choosing this solution and not using Hiera or other tools was because most of them would only change the way the secrets are stored but not how they are provisioned, so you wouldn’t cover the case where someone is able to snoop on your network and read the secrets while in transit.

Requirements for using Deferred functions

First you should know that deferred functions are available only for Puppet 6.x and above, which means that you will have to perform an upgrade if you have a lower version like we did.

One big change that affected us was the removal of mcollective which we’re using to run queries on multiple nodes. A workaround that worked in our case was to fetch mcollective server and client from the Puppet sources and install it manually using the scripts provided by Puppet in the 5.x version.

Another change is that Facter was updated from 3.x to 4.x and some of your variables may have different names (pretty small chance) but for us it was fine.

Finally the last big change that could impact you is the deprecation of the ruby functions written on top of the legacy 3.x API, all your modules that still use the old API will have to be refactored. If you’re lucky and the module that you use is updated you can fetch the latest version if it’s still compatible with the already provisioned code or change them on your own following the official guide.

The best part about newer Puppet versions is that they’re compatible with each other if you’ve got at least 5.x, for example you can upgrade the server to 6.x/7.x and slowly roll the agents as you refactor things since any 5.x agent will be able to connect and work with a newer version of the master. This really helped us move faster and not have too much of an impact on the current provisioning flow.

More about Deferred

The Deferred type is a special type that takes the function name and the values as arguments which will be used to call that function in the future (aka on the agent at runtime instead of doing it when compiling the catalog).

Let’s assume we want to echo the current time:

This will echo the current timestamp at which the catalog was compiled on the master node.

Changing it like so will make it defer the call to the agent and echo its timestamp instead.

Using Deferred functions to provision secrets

Following the idea from above we can use the Deferred type to delegate the fetching of the secrets from Vault to the agent nodes, and have each agent use its own credentials, thus giving us very fine control on who has access to what secret.

We took inspiration from puppet-vault_lookup but decided to rather write our own function with a different authentication method. We were already using the IAM Auth flow to get Vault credentials which can be later used to fetch the secrets so it only made sense that we use it in this tool also.

Each instance has a special IAM Role assigned to it and these roles are bound to some Vault roles that allow you to fetch secrets from certain paths.

For example an instance from the monitoring group will have access only to monitoring secrets, if we need even more granular control we can create a role for each instance but that would be overkill.

Refactoring the code to use Deferred functions

Before having the vault-lookup function and Deferred functions we would simply declare the secrets in the init.pp and render the templates. This is how it looked..

Initially the template uses the values from our class and after it is rendered on the server it sends it to the agent, but this means any agent can have access to these secrets and the master is node is like a SPOF for our secrets. Let’s use Deferred to make this a bit more safe.

First we will need to fetch the secrets from Vault. We do that by replacing our variable initialization.

What will happen now is that the agent will have the responsibility of executing the code for fetching the secrets from Vault and only that instance will see those secrets.

But this means that we also have to defer the template rendering to the agent since the master node doesn’t have access to them anymore. A downside of this is that if you were not sending all the arguments specifically to a template, you will now have to add them to the $args map and send them.

What happened with the template is that we first converted it to eppinstead of erband we also deferred the rendering to the agent. Deferred only works with inline_epp for rendering and the function expects the template content itself, not a path to a file so that’s why we used thefind_template function.

Some caveats that we found while using Deferred

• It doesn’t work with bson, so if you try to use it with binary data and the catalog is sent as bsoninstead of json none of the Deferred functions will be executed.
• You need at least Puppet 6 on both the agent and master nodes to use Deferred.
• It doesn’t work with erb and you have to convert all your templates to epp.
• When deferring the rendering of a template you must sent all the variables used inside the template as arguments.

Conclusions

We encountered some issues while doing the Puppet upgrade, most of them were related to the deprecation of mcollective and figuring out the structure of the ACLs in Vault. Another thing that posed a challenge was upgrading Puppet on all the instances while refactoring the legacy functions from our modules. But in the end it was worth the effort. The process of adapting Puppet deferred functions took around 2 months because we had to refactor lots of templates and code and debug some issues that were not so obvious from the logs provided by Puppet.

About me

I’m a Software Engineer with a passion for coding born from the dream of making video-games. Throughout my career I’ve switched from different roles in tech companies, ranging from game making to frontend and backend development and lastly operations. Occasionally you can find me doing hackathons. Whenever I’m free I enjoy studying motorsports, participating in online races and designing or modeling things to 3d print.