Puppet / Hiera: Making Puppet even more manageable

By Steven J. Vaughan-Nichols

We can all agree that Puppet is a great DevOps tool. Organizing Puppet’s data and code, that’s trickier. Fortunately, that’s where Hiera can help.

First, you need to understand that Puppet manages dozens to thousands of servers by providing configuration management services with a client/server approach. Your managed servers, called Puppet agents, get their configuration profiles from the Puppet master.

In turn, the Puppet master holds the agents’ configurations in its PuppetDB and Hiera data stores. Puppet’s instructions are written in the domain-specific language: Puppet. This language is remindful of JavaScript Object Notation (JSON), but it’s written in Ruby. You can also use native Ruby within Puppet programs.

The Puppet language is not a shell language, such as those used in Windows’ PowerShell or Unix and Linux’s Bash shell. Nor, is it a full-fledged language like PHP. Instead Puppet uses a declarative, model-based approach for IT automation.

Puppet models everything — the node’s current state, the desired end state, and the actions needed to move from one to the other — as data. Each agent “receives a catalog of resources and relationships, compares it to the current system state, and makes changes as needed to bring the system into compliance.”

It’s that last part — treating data as part of the code — where Hiera proves its worth. As any developer knows, separating code and data is almost always difficult. When it comes to DevOps, it’s helpful to design a data hierarchy, which can cascade through server classifications.

As Scott Lackey, a senior sysadmin for Clicktripz wrote, the primary value that Hiera provides is the ability to separate the code for “how to configure the /etc/ntp.conf” from the values that define “what ntp servers each node should use.” In short, Hiera lets you separate the “how” from the “what.”

Hiera does this by providing a key/value lookup tool for configuration data. This data is kept in a simple, pluggable hierarchical database. Hiera can search through all the tiers in this hierarchy and merge the result into a single array. This enables you to keep site-specific data out of your manifests. This, in turn, makes your Puppet modules much more portable. Typically, Hiera data sources are written in easy-to-read YAML.

Hiera’s typical hierarchical lookup uses a “defaults, with overrides” pattern. This means you specify default common data once, then override it when the default won’t work. Hiera uses Puppet’s facts to specify data sources, so you can structure your overrides in whatever makes sense for your infrastructure.

According to Puppet — and Spider-man, Hiera is immensely powerful, and with great power comes great responsibility. Specifically, you must make your infrastructure consistent and readable.

What does that mean? You should adopt sensible, rigorous rules about where and how Hiera data enters your system. You can, of course, write terrible data rules in Hiera, but you can also start with Hiera’s roles and profiles method about what should and shouldn’t be configured with Hiera.

Puppet claims Hiera is the most flexible way to get configuration data into Puppet. I agree. I can’t imagine using Puppet without Hiera.

The Puppet team agrees, too. Starting with Hiera 5, it’s now built into Puppet 4.9. This version was released in February 2017. Hiera 5 is still not in Puppet Enterprise, the commercial version of Puppet. The latest Puppet Enterprise edition uses Hiera 4.10. Puppet Enterprise will be upgraded soon.

Hiera 5 is backwards-compatible, so most of you won’t have trouble moving up to 5. The key phrase is “most of you.” There are some Hiera compatibility issues you may need to address.

The program’s biggest new feature is its independent hierarchy configurations for each environment and module. This means:

• Your main Hiera data and its configuration now coexist. This means making changes to the hierarchy is as safe and testable as any other change to your code or data.
• Module authors can use the power of Hiera to set default values for their modules, and users can override defaults without worrying over implementation details.

Hiera 5 also boasts an overhauled interface for building custom backends. This makes it possible to integrate Hiera with almost any data source.

Put it all together and what do you get? A DevOps package that makes setting up servers easy, and, once done, easier still to migrate from project to project.

Please feel free to share below any comments or insights about your experience with or questions about Puppet and Hiera. And if you found this blog useful, please consider sharing it through social media.

About the blogger: Steven J. Vaughan-Nichols is a veteran IT journalist whose estimable work can be found on a host of channels, including ZDNet.com, PC Magazine, InfoWorld, ComputerWorld, Linux Today and eWEEK. Steven’s IT expertise comes without parallel — he has even been a Jeopardy! clue. And while his views and cloud situations are solely his and don’t necessarily reflect those of Linode, we are grateful for his contributions. He can be followed on Twitter (@sjvn).