Testing Ansible Automation with Molecule Pt. 2
Do more with Molecule to make sure your infrastructure works! Use linting, idempotence, multiple containers, and backend dependencies to allow you to deploy a website while confirming your Ansible roles behave as they should.
In the first blog post, we looked at the basics of setting up Molecule and running some simple tests. In this post, we will dive deeper into the molecule configuration and various checks it can perform.
This post will explore running two distinct containers, validate idempotency, and check syntax with linting. You can find the Ansible code we are starting with on Github. Feel free to follow along or look at the completed code here.
The Role
The role we will use is a practical example that can reflect a real-world scenario but should not be considered production ready. With that said, let’s take a closer look at what it does.
The function of this role is to deploy a Laravel website. It will install Nginx, PHP 8.1 (and packages), and Composer; from the role will build the Laravel site. Lastly, we will use Molecule to deploy containers, validate the database migration tasks have successfully completed, confirm idempotency, and ensure our code meets linting standards.
Tasks
We can start by looking at the various task files.
From here, we will use specific files to install key web components (Nginx and PHP) and deploy the Laravel website.
The tasks to install Nginx are simple: update the apt cache, install some packages, start the service, and set the configuration. The critical thing to point out here is on line 5 cache_valid_time: 3600. This option tells Ansible not to run apt update if the cache is newer than 3600 seconds, which is essential for testing idempotency (note: we will cover idempotency later in this post). This task file also drops off the Nginx configuration file for the website.
The tasks in php_install.yml will install PHP 8.1 and the various packages needed for the website. After installing PHP, we will also set up Composer. The packages list and the composer_path are in a list in defaults/main.yml.
The last set of tasks deploys the code for the Laravel site. I am using a project created by Jeffrey Way, Laravel From Scratch Blog Project, which will create a local blog site when deployed using Molecule.
The above tasks will clone the project to our host, which will create both the .env and the databse.php files. We will set the environment variables needed for the .env file inmolecule.yml. Next, we update the Composer dependencies and run an install. Lastly, we use Artisan to run migrations, seed the database, and generate the APP_KEY to secure our site.
That is where we will start with our role. Keep in mind that we aren’t testing the website itself with Molecule, we are just validating the ability to deploy the site. And as we investigate idempotency and linting, I will update these files accordingly. But, before we get into linting, let’s look at how we run migrations or seeds without a step to install a database. To do that, we need to look at our Molecule configuration and how to use multiple containers.
Molecule
Platforms: Multiple Containers
One of the benefits of Molecule and Docker is that it provides a means to deploy multiple containers, which means that if we want to test a database or messaging role, we can. In our case, it allows us to deploy a MySQL container, run migrations, and seed data for our Laravel role.
Now let us look at the molecule.yml file. Similar to what we did in part 1, the platforms: block is where we want to configure our containers. This section allows us to configure containers similarly to docker-compose. Here we are defining networks, volumes, images, and open ports. For this test suite, we are leveraging containers created by Jeff Geerling. These containers are specifically designed to allow the testing of services deployed by Ansible in containers.
A brief explanation of some of the options I am using here:
name:this value is used as the hostname of the container for both the container network and the Ansible inventoryVolumes:this volume mount is required for the systemd service to start properlyprivlaged:this option tells the container if it should run as rootpre_build_image:this option tells Molecule to pull the container from a registry instead of building itpublished_ports:like-pfor Docker, this tells Molecule to map the listed ports between the host and the container. Meaning that once we have a successfulconvergewe should be able to go tolocalhost:8080and see our site running.networks:leverages thedocker networkcommand to create a separate network to run the containers in. In this case, the network we want is named “laravel”
It’s important to note that you can use any arrangement of images in this block. For example, we could use multiple operating systems to test a role in parallel or identical containers to test a cluster. The only thing to keep in mind is the name provided in the platforms: block as Molecule uses those to create your inventory.
Provisioner: Environment Variables
There are numerous Ansible and Molecule methods for handling environment variables. I chose to provide them in the provisioner block for two reasons. One, it provides clarity for reading molecule.yml since everything is in one place. Two, since we are deploying more than just our Laravel role with Molecule, I wanted one place for values that are accessible by both the site and the database.
Converge: Deploying two roles
At this point, we can deploy two distinct containers on a shared network and provide Ansible with some environment variables. So how do we go about provisioning these distinct containers? There are a couple of key pieces needed for this. We can start by adding dependencies using Ansible Galaxy and a requirements.yml file. Then we can add a dependency: block to our Molecule file.
These pieces tell Molecule that we need to install my fork of the geerlingguy.mysql role from GitHub before we process our converge. Now that we have the role that will install MySQL onto our container, we need to run it. We should take a look at our converge.yml file. In my previous post, I said that the converge.yml is the playbook that Molecule will use to create our containers. To that end, our converge will need to target multiple hosts. I mentioned earlier that the name we provide each container in theplatforms: block is what Molecule uses for inventory. Allowing us to do something similar to this:
At this point, our molecule.yml should look something like this:
Finally, all of the prep work is complete, and we should be able to run the command: molecule converge . If everything is working correctly, Molecule should pull the MySQL role from Galaxy, launch two containers, and run the playbooks against their target hosts. We are publishing ports on the containers, so we can go to http://localhost:8080 and interact with a functional blog.
Idempotency?
Yes and no.
Yes, we have confidence that tasks that rely on different hosts can do what we want.
No, because a number of those tasks will fail when run against an existing server. These failures could result in headaches for developers or, even worse, downtime in production.
Let’s start by defining idempotence. According to Wikipedia, “Idempotence is the property of certain operations in mathematics and computer science whereby they can be applied multiple times without changing the result beyond the initial application.” Meaning no matter how many times we run an Ansible playbook, it should only apply changes the first time a task is run.
That is why idempotency is vital for your Ansible roles. Why install something twice? Don’t risk updating the running version of Java or Nginx when you can use Molecule to identify things that might change if you run your role twice.
When we run the idempotence command, Molecule will rerunconverge.yml and check that none of the tasks create a change. Let’s see what that output would look like. Run the command molecule idempotence and you should get the following:
We can see that several tasks are making changes to the running instances. We can also see that the geerlingguy.mysql role is causing our idempotence check to fail, but since we are not testing that role, we need to ensure that our idempotence check ignores that part of the converge. To do that, we can add the line tags: molecule-idempotence-notest.
Like most Ansible tags, we can use this in any scope. Here we are using it to skip an entire role, but we could also use it to skip tasks. With that update, let’s rerun the idempotence check. Molecule will now ignore the first section of the converge that targets the database container and give us output that looks like this:
Fantastic, no MySQL host in the output. Now that we know our Molecule checks will only target what we want, we need to clean up the rest of our code. The benefit of the output is we know which tasks, exactly, are causing our idempotence issues. We will start at the top with the task that installs Composer.
Since we are using the command module to check if the Composer binary is present, it will always return changed. What we care about is the output we register, so we can simply add the changed_when: false option to the task Checks if Composer is already installed in the php_install.yml file.
The rest of the idempotence failures are coming from deploy_site.yml. If we look, we are executing several tasks that are only required when initially deploying the site. To fix that, we can wrap these tasks in a block with a when statement that checks if the git clone occurred. First, we want to move the Clone git repository task from the existing block to a stand-alone task and register the task’s output. Remember, we will need to add the options to run the task as the userwww-data to lines 10–17. Once we have moved that task, we want to modify the remaining block to check the registered output for the changed status on line 56.
Once we make those changes, we can rerun the idempotence test. Now, let’s run the command molecule idempotence and see what we get. If all of our code is updated correctly, we should see the following output:
Excellent, we are one step closer to a clean and functional Molecule test suite.
LINTING
At this point, we know that our Ansible code is functional and idempotent, but is it clean and up to standards? To find out, we can run our code through a linter. To test linting in Molecule, you first need to enable it in the main file. We can do that by adding a lint: block to the end of molecule.yml beneath the verifier: block.
You’ll notice that I am using two different linters here yamllint and ansible-lint. yamllint focuses on YAML syntax and other YAML-related practices, while ansible-lint focuses on Ansible-centric code and behaviors. There is a third linter I would recommend if you are using testinfra, flake8, which will help keep your Python tests up to standards.
Once we have added the lint: block, we can run molecule lint and see where we might want to improve our code.
Wow, that is a lot of potential issues to resolve. Where should we begin? Let’s start with the one that has 21 occurrences, fqcn-builtins: Use FQCN for builtin actions. This error means we aren’t using the Fully Qualified Collection Name (FQCN) for many of our Ansible tasks. We need to update all of the impacted tasks toansible.builtin.<MODULE_NAME>. To keep things simple, I am just showing the error and what the new task identifier should look like.
Whew, now that those are fixed, let’s look at what is remaining.
Of the remaining errors, there are two that we can ignore: meta-incorrect, which means the file in our ./meta directory still has the default information, and no-changed-when , which means that specific tasks are not idempotent. Since we are not concerned with the meta information and added checks and tests for idempotency, we can skip both of them. So how do we skip these checks? By adding a .ansible-lint file to the Molecule directory.
With that done, we can run molecule lint again and see that we are down to 4 violations.
First is thegit-latest error. It tells us that we must pin a version to our git task instead of assuming the mainline branch. Fixing this is as easy as adding a single line version: main to the task.
Next is the no-handler violation. This error lets us know that we have a task or series of tasks that only execute on a change. We should use a handler feature to run the tasks. Fixing this violation requires a series of changes starting with moving the confugure site block of tasks from deploy_sites.yml to a separate file. I went with tasks/configure_site.yml.
Then we need to make an additional update to the git task, adding another handler.
Lastly, we need to update handlers/main.yml to call tasks/configure_site.yml to act when the notify is triggered.
With that done, our tasks to configure our site and set up our database will now only run when the git task registers a change.
The last lint violation is risky-file-permissions on two files. In short, we didn’t specify permissions for the files handled by Ansible, leaving room for overly permissive or restrictive permissions on files. This change is easy to resolve by adding the mode: <PERMISSIONS>option to the files we are handling.
After we finish making these changes, we can run molecule lint one last time and see that it doesn’t return any violations.
Wrapping Up
Once that last section is complete, we can put it all together and run molecule test Allowing us to see all of these changes in one run.
Success!! Our Ansible role can test our database migrations, and we have successfully confirmed its idempotence and worked to ensure it meets coding standards. Now you can go forth with the knowledge and confidence that your Ansible roles will be healthy and stable.
