Benefits of Deploying Red Hat Satellite and Ansible for Patch Management
Keeping critical systems patched and updated is one of an IT team’s most important duties. However, patching servers manually, or even automating patching using advanced tools such as Ansible alone is an outdated practice that can leave your infrastructure vulnerable. A better solution is to implement a Linux patch-management system.
This article focuses on Red Hat Enterprise Linux (RHEL) patch management using Red Hat Satellite in conjunction with Ansible automation for patching and security. Red Hat Satellite Server is the recommended patch management solution for RHEL servers, which comprise 33% of the paid enterprise server operating system market, making it the leading Linux platform in the enterprise.
What is Red Hat Satellite
Red Hat Satellite Server is a system management solution that makes Red Hat infrastructure easier to deploy, scale, and manage across physical, virtual, and cloud environments. — Red Hat.
Satellite allows you to provision, configure, and update systems to ensure they are running efficiently and compliant with relevant standards.
Many large companies have thousands of RHEL servers in their data centers, all of which must be patched regularly and without causing critical outages. Satellite provides critical features to improve patching aligned with an organization’s designated lifecycle environment and needs.
Satellite provides:
- Content repositories of packages which can be made available to any host
- Distribution of content as close as possible to the end point
- Reporting on hosts that need updates, fixes, or enhancements
- Groups for systems based on criteria important to your organization
- Scalable automation to respond quickly to patching requirements
One benefit of using Satellite versus patching servers manually or with other automation methods is the ability to use lifecycles, which automate the patching schedule and process.
For example, you could set up a development (DEV) and production patching lifecycle that sets Satellite server to sync RHEL content directly from Red Hat each week. It first promotes content from the initial sync’s base library to the DEV lifecycle using Ansible playbooks within Ansible automation controller. Two weeks later, the playbook automatically promotes this content into production, giving the DevOps teams two weeks to test and validate patches against the DEV servers and verify they are safe to promote into production. This is a critical step in protecting your production environments from introducing patches that can cause unwanted outages. This method also allows you to lock repositories during the lifecycles to ensure all DEV or production servers have the same patches.
How we patch with Satellite and Ansible
At IBM, we use capsule servers to deploy patches throughout our environment. The main Satellite server is located in one of our datacenters in the United States, with multiple capsule servers within the United States and several locations worldwide. This allows patching to be as close as possible to each host. The capsule servers sync the repositories in Satellite to each region and provide the same patch level to all hosts, no matter where they reside in the world.
You can use Ansible automation controller or Satellite’s built-in Ansible functions to automate the entire server patching process with playbooks.
At IBM, we use both Ansible automation controller and the built-in Ansible playbook feature for patch management. First, a playbook in Ansible automation controller scans all servers in the environment to identify the RHEL servers. Next, the playbook determines if the RHEL server is registered properly. If it is, nothing is done. If not, the server is unregistered and cleaned, and the local capsule server installs the proper Katello certificate package, which automatically configures the host, and installs subscription manager along with its dependencies if they are missing. Then the server is registered using the appropriate activation key from Satellite. The playbook looks at which version of RHEL is installed and how the server is classified in Service Now to determine if this server should be in the DEV or production lifecycle. This playbook runs monthly to ensure all servers remain in compliance with registration.
After running this patching automation for a few weeks, we found several standard issues that caused patching to fail in our environment. So, we developed another playbook, this time using Satellite’s internal Ansible, that validates there is enough space on the server to accommodate a kernel patch. We run this playbook the day before a patch window, and if it finds there is not sufficient space, it cleans out older kernels. It then checks to see what repositories are active on the server. If it finds one that is not one of the approved repositories required for proper server operations, then it renames the repository to disable it. This prevents broken repositories from causing the entire server to fail during the patch window. These two automated tasks significantly reduce the operations team’s time spent on manually resolving patching issues with servers.
The final stage of our patch cycle is the actual patching playbook, also done with Satellite’s internal Ansible. We schedule the playbook to run ahead of time against a host collection, a group of servers that are set to be patched. There are multiple groupings, depending on if they are DEV or production servers and on their patch window’s scheduled day and time. The playbook validates the server is ready, performs any last-minute cleaning activities, and deploys security and bug-fix patches on the servers in the group at the specified time. Afterward, the playbook checks to see if a reboot is required; if so, it initiates the reboot and reports back once the server is online.
Integration with Red Hat Insights
Satellite also integrates with Red Hat Insights, which uses advanced analytics to help detect risks within an environment before they can become a problem. Insights identifies security risks on a server and provides instructions on how to resolve the issue. Many of these issues can be resolved with the help of Ansible playbooks written to mitigate the security findings. And you can do all of this through the usual Satellite interface.
Results of automating patches with Satellite and Ansible
When we started using Satellite instead of patching servers using only Ansible playbooks, we noticed a dramatic decrease in the time needed to accomplish a task. Previously, patching took the operations team an entire weekend, and now it takes only two hours. This is due to Satellite’s advance work to make sure servers are registered properly and resolve most known failures ahead of time. We also found that having seven different capsule servers patching hundreds of servers at the same time greatly improved our overall patching efficiency. Initially, we had a 95% success rate across about 3,500 RHEL servers around the globe. This percentage rose with each successive patch cycle as we resolved one-off issues on servers that had patching failures.
Another key benefit of automation and using Satellite is now the operation team touches far fewer servers during the patch window. Not only does this make their job easier, but it also reduces problems as there is less manual interaction (and therefore fewer human-caused errors) with the servers.
Conclusion
Having a properly configured patch management system in place is critical to keeping your RHEL server infrastructure patched and maintained in a timely manner. Because Satellite is included if you have RHEL entitlements with Smart Management, you may be able to deploy this at little to no cost.
If you are interested in learning more, visit Red Hat for further information on Satellite and how to deploy it in your infrastructure.
Ivan Windon is a Lead DevOps Engineer at IBM based in RTP, NC. The above article is personal and does not necessarily represent IBM’s positions, strategies, or opinions.
