AWS System Manager: A Way to Automate Linux EC2 Instances Patching

Published in

Globant

9 min readApr 14, 2023

To keep our EC2 server’s packages updated with the latest version and secure from vulnerabilities, we need to perform patching activity regularly. Performing this activity manually for numerous servers takes a lot of time. AWS Systems Manager Patch Manager can help in automating this task.

Patch Manager is one of the AWS Systems Manager Patch Manager features we can use to automate the patching of Linux and Windows instances. This Patch Manager can also scan instances and install missing patches. EC2 tags can be used to install patches on a single instance or multiple instances.

This article will show how to use Patch Baselines and Patch Groups in Patch Manager for patching activity. Patch Baselines has a list of all the critical patches, as well as a list of approved and rejected patches. Patch Groups are used to organize your instances in the System Manager for patching. I will demonstrate how to use the Maintenance Windows task to install patches using the Patching scheduling activity.

Prerequisites

Before automating the patching activity using System Manager, we must complete the following prerequisites:

Install AWS Systems Manager Agent (SSM Agent) on an EC2 instance.
Create an IAM role for the instances.
Create an IAM role for the Maintenance Window to execute the tasks.

We will start by doing the SSM agent installation and creating the IAM roles to complete the prerequisites.

Agent Installation

These are the steps needed to perform the SSM agent installation. First, connect through SSH to your EC2 instances and run the following command on the instances to download and install the SSM agent RPM package:

sudo yum install -y https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_amd64/amazon-ssm-agent.rpm

After installing the agent, check if it is running:

sudo systemctl status amazon-ssm-agent

If the agent is not running, start it:

sudo systemctl start amazon-ssm-agent

Follow this link to manually install the agents on other Linux instances.

IAM Roles Creation

These are the steps needed to perform for the IAM roles creation, so instances have the necessary permissions to connect to System Manager and execute tasks during the maintenance windows.

Create IAM role for instances

Create an IAM role and add the following permissions to the role, then attach that role to the instance. If any other role is already attached to the instances, add only the following permissions:

AmazonSSMFullAccess
AmazonSSMManagedInstanceCore

You can add other permissions if needed.

Create an IAM role for the Maintenance Window to execute Tasks

Create a role for the maintenance windows that allows them to interact with other AWS services. The “AmazonSSMMaintenanceWindowRole” policy should be attached to that role to perform the patching on the registered targets. Follow this link for more details and guidance.

Note: System Manager provides the “Quick Setup” option. You can use it to install an agent and create an IAM role for your multiple instances simultaneously.

Now, we will see how to run Patch Manager for the patching activity. In the AWS Console, open the System Manager service and click on the Patch Manager option:

Patch Baselines

Patch baselines define which patches need or do not need to be installed on EC2 instances. Patch Manager has two types of patch baselines: Predefined and Custom Patch Baselines. Predefined baselines are available for each operating system supported by Patch Manager. A predefined patch baseline defines all the patches classified as critical or security updates, with the severity of “Critical” and “Important” patches. The patches with the Classification “Bugfix” are also auto-approved. These patches auto-approve 7 days after being released or updated.

Below is the list of predefined patch baselines for multiple operating systems:

These are specific for Amazon Linux 2:

**Predefined patch baselines for Amazon Linux 2**

Create your patch baseline

You can’t customize predefined patch baselines provided by AWS. Instead, you can create your own custom patch baselines. Here, you can choose from the following categories to auto-approve your required patches for the custom patch baselines:

Operating System: Amazon Linux2, Centos, Debian, macOS, Windows, etc.
Products: e.g AmzonLinux2, AmzonLinux2.0.
Severity: All, Critical, Important, Medium, Low.
Classification: All, Security, BugFix, Enhancement, Recommended, NewPackages.

Use the above configurations for the Approval Rule. You can create multiple rules for the custom patch baselines. For each rule, you can specify an auto-approval delay in days or set a specific date. For example, if you set a delay of 6 days, then new patches that are released on December 1st, will be auto-approved on the 7th of December.

Make your patch baseline the default

The predefined patch baselines are the default baselines, but you can make your custom patch baselines the default while creating them or later. To set a new rule as the default rule, check the “Set this path baseline as the default patch baseline for Amazon Linux instances”, under Default patch baseline:

To configure an existing patch baseline as the default, click on the “Set default patch baseline” button:

Patch Groups

Patch groups define which patch baselines will be used for which instances. For example, we can create patch groups for each environment and associate those patch groups to a set of instances per environment. Patch groups can be defined based on server functionality, for example, web servers and databases, etc. Patch groups can help you avoid deploying patches to the wrong EC2 instances.

To create a patch group, you can click on the Patch Manager menu option and then select the patch baselines you want to include in a Patch Group. For example:

Steps:

1. Select Patch baselines “ “WebServer-AmzonLinux2-PatchBaseline” and, in Action, click on “Modify Patch Groups”.

2. Add Patch Groups values “Prod-AmzonLinux2-WebServers”.

A fleet of instances that have these patch groups tags will be patched using these patch baselines:

Maintenance Windows

Schedule and implement maintenance tasks on the registered instances. For that, we can use maintenance windows in System Manager. Maintenance windows can have a schedule, a duration, a window start and end date, a timezone, a set of registered targets, and a set of registered tasks. If you want to apply some patches on your application instances at a particular time when there is less traffic and the least impact on your application, you can use a maintenance window.

Create maintenance windows

In AWS Systems Manager, go to Maintenance Windows and click on “Create Maintenance Windows” to schedule patching tasks for EC2 instances. In the schedule, add the following details:

Select the CRON/Rate expression option. You have two other options as well, and you can select any option as per your requirement.
CRON/Rate expression: For Example, cron(0 30 20 ? * TUE#2 *) for 08:30 PM on the Second Tuesday of every month. Please follow this link to know more about cron expressions.
Duration: The duration in hours of the maintenance window.
Stop initiating tasks: The number of hours before the end of the maintenance window that the System Manager stops scheduling new tasks for execution.
(Optional) For the Window start date, specify a date and time.
(Optional) For the Window end date, specify a date and time.
(Optional) For the Schedule timezone, specify the time zone as per your requirement.

Register Targets

Go to the maintenance window created before, then go to Targets and click on Register target. Specify the patch groups tag for the tag name and any value for which you have defined EC2 tags, (for example, Prod-AmazonLinux2-WebServers). You don’t have to target patch groups, instead, you can use any tag defined for your instances, completely independent of the patch groups tag:

Now, add the tag details to the web servers on which you want to perform the patching activity in the setup maintenance window:

Register Tasks

Go to Tasks and click on the Register tasks combo to add tasks to the maintenance window:

1. Select “Register Run command task” from the list:

2. Add a name and description of the task:

3. In the Command document, select the document AWS-RunPatchBaseline for patching tasks. This command document can be used for Windows, Linux, and macOS.

4. Select the document version to use and the task priority that defines the order in which tasks run when multiple tasks are registered:

5. Select the Windows target ID, which we defined earlier:

6. The Rate control option specifies:

Concurrency: to Allow targets to run tasks at one time, we can specify either an absolute number of nodes, for example, 15, or a percentage of the target set of nodes, for example, 20%.
Error threshold: The number of errors that are allowed before the system stops running the task on the next targets. We can specify either an absolute number of errors, for example, 1, or a percentage of the target set, for example, 10%.

7. In the IAM service role, Select from the list the role that we created in the prerequisites:

8. In the Output options, there are two options: Write to S3 and CloudWatch output. Using these options, you can store your command execution output. Here I am using the Write to S3 option:

9. The Parameters section specifies:

Operation: can be either Install or Scan; you can choose as per your requirement.
Reboot Option: it can be RebootIfNeeded or NoReboot; you can choose any of these options for your targets after the patching activity.
Timeouts(Seconds): Default It is 600 Seconds.

10. Now, click on Register Run command task.

That’s it! Now the maintenance window will execute as per your scheduled time and it will perform the patching activity on the registered targets.

Manually enable or disable the maintenance window

You can configure the maintenance windows to enable or disable them:

**Enable and Disable maintenance windows**

Conclusions

In this article, we have learned about the Patch Manager feature in AWS System Manager. We have covered the key aspects of System Manager, like SSM Agent, patch baselines, patch groups, and maintenance windows.

We started by showing you how to install the SSM Agents on Linux instances, and then how to create patch groups and patch baselines that define which patches should and shouldn’t be installed on your instances. After you defined your baselines, we finally saw how to register tasks and targets for the maintenance windows.

This will greatly reduce manual patching efforts by using a maintenance window and scheduling the rollout of patches.