Foreman and Puppet in the Infrastructure as Code (IaC) approach.

12 min readOct 17, 2023

· Introduction
· Puppet Manifests in the Foreman Web Interface
· Puppet Code and Puppet ENC
· Foreman Group hosts, Organization, Location and Parameters
· Viewing Puppet facts in Foreman.
· Report templates in Foreman
· API Foreman
· Conclusion

Introduction

This is an introductory publication providing an overview of some features of the Foreman platform. They relate to Puppet in the context of Infrastructure as Code (IaC).

Foreman offers a lot of tools to control your IT environment, but this article specifically focuses on certain capabilities such as:

Using Puppet modules
Configuring parameters for Puppet
The module deployment process
Writing scripts for collecting the required facts
Utilizing templates for generating reports
Exploring the use of APIs and databases

The capabilities of Foreman are described from the perspective of a platform user. The technical implementation of the demonstration setup will not be covered.

If you are interested in reading about the installation of Foreman, the Bare Metal Controls plugin, setting up GitHub Actions for Puppet code CI, using r10k, Puppet code linters and automated Puppet code testing, as well as working with Smart-proxy (PXE, DHCP, DNS), please contact us. Our contact information is provided in the publication’s conclusion.

Infrastructure Components:

+==========================+=====================================+
|          Server          |        Installed Components         |
+==========================+=====================================+
| foreman-main.example.com | Foreman, Puppet Server, PostgreSQL  |
+--------------------------+-------------------------------------+
| test-server.example.com  | Puppet Agent                        |
+--------------------------+-------------------------------------+

We are using the Ubuntu 20.04 LTS operating system and Foreman version 3.7 without any additional plugins.

Puppet Manifests in the Foreman Web Interface

Task:
Install Zabbix Agent and specify the Server address as 8.8.8.8 in the agent’s configuration.

Possible Solution Using the Foreman Interface:

Step 1: From the list of hosts, select “test-server” and click on “Edit.”

Step 2. In the Puppet ENC tab, assign the Zabbix::agent manifest.

Step 3. Set the value for the Server parameter and click Submit.

Step 4. After applying the Puppet Agent configuration, we check the report for successful installation.

In the latest report, charts of manifest application are displayed. If the package has already been installed, the configuration will be modified and the Zabbix agent will be restarted.

Step 5. To verify the application of the configuration in the server console, search for the Zabbix Agent package and find the new value of Server in the zabbix_agentd.conf file.

Here it is, the task is complete! Next, the Puppet Agent will check and maintain the specified configuration every time it connects the Puppet Server.

In this example and throughout the article, we will be using the excellent Zabbix Module taken from https://forge.puppet.com. However, you can create Puppet modules according to your own preferences and ideas.

In the following chapters, we will explore overriding parameters used in Puppet code.

Puppet Code and Puppet ENC

Development of Puppet code is no different from any other development and can be configured using version control systems and your favorite IDE. This publication uses Pycharm and GitHub. We already have Pycharm set up and a GitHub pipeline for delivering code to the Puppet Server.

Task:
Override the default parameter responsible for the network address of the Zabbix Server in the Puppet module Zabbix.

Step 1. In the Zabbix module, edit the params.pp manifest file, change the value of $agent_server from “127.0.0.1” to “8.8.8.8”, and send it to the repository.

Step 2. Run CI. Code is delivered to Puppet Server, the same as Foreman.

If necessary, in the console of the foreman-main server, check that the modules have been delivered and the parameter has been correctly set.

Now, by default, the address Server=8.8.8.8 will be set for all newly installed Zabbix agents.

You are able to override the value of the Server parameter for one host using Puppet ENC.

Step 1. Specify the new value for the parameter for the test-server host.

Step 2. After applying the Puppet Agent configuration, we review the changes using the “Show Diff” button, where the agent’s work details are displayed: the modification of the zabbix_agentd.conf configuration file line.

Wonderful capabilities, so you can easily deliver the code to Puppet Server → Puppet Agent and use default parameters in Puppet code, as well as override them in Puppet ENC.

Below we will show the implementation of default and overridden parameter values through Foreman in Puppet ENC.

In the left part of the IDE, the params.pp manifest is open with the default value of the $agent_server parameter (also known as server in the Foreman interface).

In the bottom part of the IDE, the connection to test-server via ssh is displayed, along with the value of the Server parameter from the zabbix_agentd.conf file.

In the right part of the IDE, there is a connection to the Foreman database and the contents of the lookup_keys and lookup_values tables.

Summary:

For all servers where the Zabbix agent will be installed, the $agent_server parameter is set to ‘8.8.8.8’.
For test-server.example.com, the $agent_server parameter is overridden in the database and set to “7.7.7.7”. Priority will be given to the value specified in the database.

Next, let’s consider the abstractions in Foreman: Host Groups, Organizations, and Locations, and the parameters that can be assigned to them.

Foreman Group hosts, Organization, Location and Parameters

Foreman system has a flexible logic of host grouping and parameter nesting. Thanks to this, it’s possible to control the configuration based on the server’s functional purpose. For example, you can separate hosts depending on the required software, server locations, and their assignments.

It’s not possible to cover it comprehensively in one publication, but let’s consider the simplest scenario.

Task:
Сreation of a group that, when assigned to servers, will install Zabbix Agent with the specified parameter $agent_server, different from the default value.

Steps to solve the task:

Create a host group.
Override the parameter in the Puppet ENC for the created host group.
Assign the group to the server test-server.example.com.

Step 1: Create a group, specify the mandatory fields, and set the Puppet Environment. In this case, it is the name of the “production” branch from the repository. You can have as many environments as there are branches in the repository, and different groups can be assigned different environments.

Step 2: Set Puppet ENC parameters for the group.

Step 3: Clear the previously set parameter and move our server to the new group, “test_group.”

The assigned group is displayed in the main host interface.

After applying the Puppet Agent configuration, we can see a new parameter in the Zabbix agent’s configuration that has been overridden for the host group “test_group.”

Now let’s consider the possibility of creating group parameters. They are different from Puppet ENC parameters and can be used in Puppet modules.

Assign the “test_group” a new group parameter, “test_parameter_zabbix,” with a data type of string and a value of “5.5.5.5.”

We use the group parameter “test_parameter_zabbix” as the default value for the variable $agent_server in the Puppet manifest. Make sure to remove the override from the Puppet ENC tab beforehand.

You can find the discussed parameter in the “parameters” table.

After applying the Puppet Agent configuration, the report displays the value set using the group parameter.

Similarly, you can manage hosts with divisions into Organizations and Locations. They can include Groups and individual hosts.

Let’s group servers into two branches: North and South. These abstractions will have different values for the $agent_server parameter.
Now, let’s look at the creation of the “North Branch” organization.

Step 1: In the Organizations menu, click the “New Organization” button and provide the name.

Step 2: Specify the use of Smart Proxies.

Step 3: Set a new organization parameter, $branch_parameter_zabbix, add it to the Zabbix module, and add the server to the organization.

Specify the created organization parameter in the params.pp manifest, which can be viewed in the “parameters” table.

After applying the Puppet Agent configuration, the report displays the value set using the organization parameter.

In this chapter, we’ve explored the ability to catalog servers and use parameter hierarchies. You can learn more about parameter hierarchies by following this link.

Next, let’s move on to Puppet facts and how to view them.

Viewing Puppet facts in Foreman.

In this section, we will learn how to view Puppet facts in the web interface and in the database.

Puppet collects information about servers in the form of abstractions called Puppet facts.

There are default facts, such as the OS name and network address. It’s also possible to collect custom facts, such as the presence of mounted network directories, the version of installed software, and more.

One way to view collected facts is through the “Monitor → Facts” tab in the web interface. You can search for facts based on host attributes, fact name, its value, and the presence of a fact.

For example, let’s request all collected facts for the host test-server.

There’s also the option to access facts directly from the database.

An example SQL query to retrieve facts:

select distinct hosts.name, fact_names.short_name, fact_values.value
from hosts
join fact_values on hosts.id=fact_values.host_id
join fact_names on fact_values.fact_name_id=fact_names.id
where hosts.name  = 'test-server.example.com';

The screenshot shows the execution of the query in the Foreman database, resulting in the retrieval of facts for the host ‘test-server’.

With this information, you can visualize data using Grafana, perform analysis, and discover patterns.

Task:
Obtain data about NFS mount points on all servers in the server park.

This task can be solved by using custom facts. Their values can be utilized in the logic of Puppet manifests.

Here’s an example script for collecting a custom fact about mount points. This script will be executed on all servers, and the output will be obtained by the Puppet Server, making it accessible in the database and the web interface.

# nfs_search.rb

awk = "/usr/bin/awk"

if File.exist?(awk)
    Facter.add('nfs_search') do
        setcode do
          Facter::Core::Execution.execute('\'awk\' \'$1 ~ /^([0-9]|\/\/)/ {print $1 ","}\' /proc/mounts')
        end
    end
end

In this chapter, we’ve explored the ability to collect information about servers using Puppet facts. These values can be used in manifest logic and system audits.

Next, we will look at using the collected facts in report templates.

Report templates in Foreman

Report templates use the ERB templating engine to create reports that can be generated as files through the web interface or requested using the Foreman API. Let’s explore the use of such a report and how to retrieve it from the web interface. Reports can utilize the collected facts.

Task: Obtain a list of all hosts, their network addresses, and the names of their operating systems.

Step 1: In the menu, select “Report Templates,” and use the previously prepared template.

Step 2: When selecting a template, you can optionally specify a search expression. By default, all hosts registered in Foreman will be included in the report. Choose the file format. After clicking the “Generate” button, the report will be available for download and viewing.

Step 3: In the generated report file, you will see 2 servers, including columns with network addresses and the OS names.

A template was used to generate the data:

<%#
name: test - status
snippet: false
template_inputs:
- name: hosts
  required: false
  input_type: user
  advanced: false
  value_type: search
  resource_type: Host
model: ReportTemplate
-%>

<%- load_hosts(search: input('hosts')).each_record do |host| -%>
<%-   report_row({
        'server_name': host.name,
        'network_address': host.facts['ipaddress'],
        'OS': host.facts['os::name'],
              }) -%>
<%- end -%>
<%= report_render -%>

Reports can be used to address various tasks, such as infrastructure audits.

All reports available through the web interface can also be requested using the Foreman API, which we will explore further.

API Foreman

In this chapter, we will explore the use of the Foreman API.

Task:
Retrieve report data through an HTTP request.

Example of solving the task using Postman:

Step 1: Specify the POST method, login, and password for Basic Auth authentication type (Foreman can use tokens for authentication).

Step 2: Specify the JSON data type and the request body: { “report_format”: “json” }

Step 3: In the server response, you will receive the generated report in JSON format based on the “test_template” template.

Such capabilities can be useful in implementing automation and integration.

Let’s solve the same task using a Python class.

#!/usr/bin/python3
import requests
from pprint import pprint
import urllib3

urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)


class ReportGenerator:
    def __init__(self, api_url, auth):
        self.api_url = api_url
        self.auth = auth

    def generate_report(self):
        json_data = {
            "report_format": "json"
        }

        response = requests.post(self.api_url,
                                 auth=(self.auth['user'], self.auth['pass']),
                                 json=json_data,
                                 verify=False)

        return response.json()


if __name__ == "__main__":
    auth_credentials = {'user': 'your_login', 'pass': 'your_pass'}
    api_url = 'https://foreman-main.example.com/api/report_templates/148/generate'

    report_generator = ReportGenerator(api_url, auth_credentials)
    api_response = report_generator.generate_report()
    pprint(api_response)

The script execution result:

The Foreman API is highly functional and can be a valuable tool for automation. In this publication, a simple example of exporting a Report Template was demonstrated.

API capabilities include:

Creating/deleting hosts
Revoking certificates
Parameter overrides at any level
Assigning Puppet manifests, groups, organizations, and much more

You can learn more about all the API capabilities on the official website at https://www.theforeman.org/.

Conclusion

Utilizing the Foreman and Puppet platform in the Infrastructure as Code (IaC) approach is an effective way to manage infrastructure. The methods used in Foreman ensure order, idempotency, security, and high flexibility in all parts of the system. Foreman complements the infrastructure landscape, significantly reduces technical discrepancies, and provides opportunities for integrations and automation.

Thank you for reading up to this point. This publication is an attempt to share user experience. We briefly touched on the functionality related to the IaC approach, trying not to overload the text with technical details.

Don’t hesitate to use comments on this article and our contact information. We are genuinely interested in your feedback. We plan to start a series of publications about Foreman and its components.

Contacts: Email address:
george.article.feedback@gmail.com

Authors of the publication:
George Minashvili, Yana Volia, Igor Cherdakov.