Part 11 — HumanGov Application — Ansible-1: Configuration Management with Ansible

The HumanGov Application is a Human Resources Management Cloud SaaS application for the Department of Education across all 50 states in the US. Check Part 10 to follow up the project series. In the first 10 part we focused on Terraform. Now, we are going to introduce Ansible and keep improving the application architecture with DevOps tools in the following sections.

In this following project series, we are going to transition the architecture from a traditional virtual machine architecture to a modern container-based architecture using Docker containers and Kubernetes running on AWS. In addition, we will also be responsible for automating the complete software delivery process using Pipelines CI/CD using AWS services such as AWS CodeCommit, AWS CodePipeline, AWS CodeBuild, and AWS CodeDeploy. Finally, we will learn how to monitor and observe the cloud environment in real-time using tools such as Prometheus, Grafana, and automate one-off cloud tasks using Python and the AWS SDK.

In the previous section, we used the Terraform to provision infrastructure with ease. In this section, we are going to provision the application to run inside the EC2 instances for each state and look at where Ansible fits in our HumanGov SaaS project. As you know from previous sections, each state has its own infrastructure. None of the resources are shared among the states. For example, each state has its own EC2 instance to run the application. Inside the EC2 instance, each state will have an application server to serve only the state it belongs to. Now, we need a way to deploy the application for all 50 states and configure the hosts in a scalable way that is prone to human errors. Instead of doing this manually, we will use Ansible to manage the configuration in a fully automated way.

Configuration Management

Configuration Management is a process of maintaining computer systems, servers, and software in a desired consistent state. It refers to the process of systematically handling changes to a system in a way that maintains integrity over time. Ansible as a configuration management tool is in use to automate repetitive tasks, ensure consistency across systems, and streamline the management of infrastructure and applications.

Configuration Management Benefits:

• Reduces the time and effort required to manage systems
• Ensures test and production environment matches
• Minimizes human errors
• Quick restoration from failures or mistakes with the ability to roll back to a previous state
• Ease of scaling such as adding new servers or resources automatically and consistently across the environment
• Enforce security policies and compliance standards across the entire infrastructure

Configuration Management: Ansible

Ansible is an open-source configuration management tool for application deployment, task automation, orchestration, cloud provisioning, security, and compliance. Ansible is agentless, meaning it doesn’t require the installation of any software. It uses SSH to connect to systems and execute tasks. It is written in a declarative language, YAML, which is easy to read and write. It also has a large collection of built-in modules to perform the tasks.

Ansible Architecture:

Ansible architecture is composed of Ansible’s Automation Engine which is software that runs Ansible. It has inventory, API, modules, and plugins inside.

• Ansible inventory file uses hosts and groups. Hosts in the inventory file specify the hosts on which Ansible connects and executes tasks. It lists the IP addresses of the target hosts. The groups can group similar servers inside a group allowing us to run Ansible commands only on these particular servers. Ansible comes with two default groups; all and ungroup. The all group includes all the hosts inside the inventory file. The ungroup group includes all the hosts that are not part of any group. You can create an inventory file in your directory. Otherwise, Ansible searches for it in the default place for the inventory file; /etc/ansible/hosts.

Ansible inventory file:

# individual hosts
[webservers]
web1 ansible_host=192.168.1.10 ansible_user=myuser http_port=80

[dbservers]
db1 ansible_host=192.168.1.20 ansible_user=myuser db_port=3306

# groups of hosts
[production]
web1
db1
lb1

• The Ansible API(Application Programming Interface) allows external programs or scripts to interact with Ansible. For example; REST API (HTTP API) and Ansible Python API. We will talk about them in detail later.
• Ansible modules are a block of reusable codes that carry specific actions, such as managing files, installing packages, or configuring services on hosts. For example, if we want to install a web server inside the host, EC2 instance, we will use a module for that. System administration and configuration modules(ping, user, cron, service), package management modules(apt, yum, homebrew), file modules(copy, templete, unarchive), user and group modules(user, group, win_user), database modules(mysql_db, postgresql_db), networking modules(ios_facts, ios_command), cloud modules(ec2, azure_tm, gcp), container modules(docker_container, docker_image), security modules(authorized_key) are the some common modules.
• Ansible plugins provide additional features, integrations, or customizations that are not provided by Ansible modules.
• Ansible playbook is written in YAML to perform configuration tasks. Key components of an Ansible playbook include hosts, tasks, modules, handlers, variables, conditionals, loops, and roles. Hosts specify the target hosts on which tasks will be executed. Tasks describe a series of actions to be performed on the target hosts. Modules are used within tasks to perform specific actions on the target hosts. Handlers are often used to trigger action tasks that are executed only if notified by other tasks. Variables provide a way to store and manage data more flexibly and in a reusable way. Conditionals allow you to control the execution of the task based on specific conditions. Loops allow you to repeat tasks multiple times. Roles are a way to organize and package related tasks, handlers, and variables into reusable components.
• Ansible facts are collected information from managed hosts during the execution of the playbook such as operation system, IP addresses, and hostname.
• ansible.cfg file is the main Ansible configuration file.
• Ad-hoc commands are CLI commands used for simple one-time tasks and use the /user/bin/ansible command line tool. We run the commands directly from the command line without using playbooks. The ad-hoc command below runs on the webserver group and module yum to install the HTTPD package on all the webserver groups.

# binary    inventory   module    module arguments
#           host/group
  ansible   webserver   -m yum   -a "name=httpd state=latest"

Lastly, Ansible is idempotent; executing the same Ansible playbook multiple times has the same result if there are no changes to apply. It performs the same action only once.

Hands-on: ANSIBLE

Part 1: Getting Started with Ansible

Step 1: Cloud9 Environment Preparation

We will keep using AWS Cloud9. Go to AWS Services and open Cloud9.

Cloud9 runs on an EC2 instance. We first need to install Ansible on the Cloud9 EC2 instance. We will create two EC2 instances on AWS. We will use those EC2 instances as managed nodes.

Go to EC2 instances on AWS and make sure you have only Cloud9 the EC2 instance running. Clean up EC2 instances and Security Groups (if any) except the ones below.

Create 2 AWS EC2:

**Instances specifications:**

- `host01`: Debian [based on Debian Linux Distribution] - 'admin' user
- `host02`: RHEL [based on Red Hat Enterprise Linux Distribution] - 'ec2-user' user

- SSH Key: 'tcb-ansible-key'
- VPC Default
- Security group: launch-wizard-1 (SSH/22)

• host01:

Create a key pair on the same page: We will use this SSH to connect to the EC2 instance. Ansible will use this key to establish a remote connection to these instances.

We will allow access to this instance from anywhere, just for hands-on purposes here. Do not do this for your production environment.

Then launch the first instance.

• host02:

For this time, we will not create a new key pair and security group. Instead, we will use the one that we created for host01.

Launch instance. Now, we have two EC2 instances created.

Step 2: Installing Python3, Pip3 and Ansible at Cloud9 control node

Before we install Ansible onto Cloud9, we first should install the prerequisites because we will use Cloud9 to run Ansible. Ansible is created in Python. Python3 should be installed on Cloud9 to be able to run Ansible. Check the Python and PIP versions on the control node (Cloud9). Python must be present across all nodes.

As you can see, Python3 and pip3 are installed already but Ansible is not. Update the OS and install Python and PIP if needed:

sudo yum update -y
sudo yum install python
sudo yum install pip

Install Ansible:

We will use Ansible documentation for everything here.

Step 3: Testing EC2 ping communication and EC2 SSH connectivity from the Cloud9 control node to the managed targets

Now, we are going to connect from the Cloud9 host to the target hosts, host01 and host02, to perform some tasks using Ansible. Before we work with Ansible we need to test ping communication and SSH connectivity. The ping checks the reachability of hosts and verifies that Ansible can communicate with them.

Go to host01 and host02, copy their private IP address, and test ping communication.

host01

host02

Instances are not answering ping requests because Cloud9 and instances are not able to talk to each other. The security group does not have a rule allowing the Cloud9 to reach the host01 and host02 instances. We need to add a security rule to the security group for that. We will allow the traffic coming from Cloud9 to host01 and host02.

Under the security, only port 22 is allowed. No internal traffic coming from Cloud9 is allowed yet.

Click the security groups, launch-wizard-1, and edit the inbound rules:

We will add a new rule that allows all the inbound traffic on host01 and host02 from the security group associated with Cloud9.

Now, we should be able to ping and connect over SSH. Let's go and check.

It is working! The same thing should apply to host02 as well because both of them use the same security group. You can stop the ping command with control + C.

Let's check if SSH connectivity is also working. Upload the private key to Cloud9 under /home/ec2-user/environment.

Cloud9 -> File -> Upload Local Files -> upload the certificate we created (tcb-ansible-key.pem)

Test the SSH connection with host01 and host02 private IP addresses.

ssh -i tcb-ansible-key.pem admin@PRIVATE-IP_host01

You can find how to connect to the host01 EC2 instance under the Connect tab as shown above. We should permit the key file first. Otherwise, we will get a warning.

We got a warning! Let's do it again.

It worked. We are in! Repeat the same process for host02 as well. host01 is a Debian instance, host02 is a Red Hat instance. Therefore, the SSH connectivity command is not the same. Our user names are different; host01 user name -> admin and host02 user name -> root or ec2-user.

chmod 400 tcb-ansible-key.cer
ssh -i tcb-ansible-key.cer admin@172.31.86.156
ssh -i tcb-ansible-key.cer ec2-user@172.31.90.116

Part 2: Ansible Inventory, Ad-hoc & Configuration

Step 1: Create an Inventory File

Open Cloud9, create a folder ansible-tasks, create an inventory file called hosts inside of ansible-tasks folder, and insert the Private IPs of the Hosts (host01 and host02) inside of the hosts file.

Step 2: Test ‘ping’ module using Ansible ‘Ad-hoc Command’

Let’s test the connectivity to the managed hosts using Ansible module ping:

ansible -i hosts all -m ping

You will see a permission denied error. We are not able to connect the hosts because we did not specify the username and the SSH key. The ping module is not only checking if instances are alive but also trying to establish SSH connectivity. It does not know how to connect without an SSH key and user name.

Let’s inform the SSH Key:

# move the key first to the current directory
mv ../tcb-ansible-key.cer .
ansible -i hosts all -m ping -e "ansible_ssh_private_key_file=tcb-ansible-key.cer"

We will still get the same error but only for one instance for this time. We are trying to connect two different hosts at the same time. The SSH key is the same for both but the user name is different. The user name is admin for the Debian instance and ec2-user for the Red Hat instance. As you can see, our Cloud9 user name is also ec2-user. Therefore, it successfully connects to the Red Hat instance with ping but the Debian instance does not.

Let's edit the inventory file. We can specify the SSH key here for all hosts instead of specifying it on the command line. We can also create a webservers group with host01:

host01 ansible_host=172.31.86.156 ansible_user=admin
host02 ansible_host=172.31.90.116 ansible_user=ec2-user

[all:vars]
ansible_ssh_private_key_file=/home/ec2-user/environment/ansible-tasks/tcb-ansible-key.cer

[webservers]
host01

ping over all hosts vs ping over the webservers group

Step 3: Ansible Configuration file

You can read here about the Ansible configuration file.

Generate a sample ansible.cfg file under the same directory and copy-paste the content of the GitHub example file content here. As an example, we can remove warnings by modifying the configuration file. Everything is commented out for now. We can uncomment related place. Change deprecation_warnings = True to deprecation_warnings = False.

As you can see, the deprecation warning disappeared!

Let's set up the ‘ping’ module as the default module and the ‘host’ inventory file path:

[defaults]
module_name=ping

[defaults]
inventory=/etc/ansible/hosts, /home/ec2-user/environment/ansible-tasks

It runs the ping module on the command line as default without specifying:

Part 3: Ansible Modules

You can read here more about modules using Ansible documentation. Always refer to the official Ansible documentation for the most up-to-date and accurate information.

First, we need to make sure that our instances are working. Run ping module commands below to check as we did before:

ansible -i hosts host01 -m ping
ansible -i hosts host02 -m ping

Step 1: Exploring the command module

You can check the command module documentation here. We will use this module frequently. Ansible will connect to the target hosts via SSH and run commands that allow the execution of shell commands on those targets.

date: The current date of the remote host

df -kh: Disk space information of the remote hosts

/etc/os-release: Retrieve the operating system information from/etc/os-release

# -i:inventory file name, -m:module name -a:argument of the module
ansible -i hosts host01 -m command -a "date"
ansible -i hosts host01 -m command -a "df -kh"
ansible -i hosts host01 -m command -a "cat /etc/os-release"

Step 2: Exploring the apt module

The apt module is specifically designed to manage packages, repositories, and package-related tasks on Debian and Ubuntu-based systems. Let's use this module to install the web server Apache to one of our hosts as an example. Before installing any package we should update the cache first. Every time we install a package or update the cache, we need to run the command as a root user for permission to perform this task.

# -become or -b is to run command as a root user
ansible -i hosts host01 -m apt -a "update_cache=yes name=apache2 state=latest" -b

We can do the same thing for many machines. This is an extremely useful and powerful way for installing and managing packages.

Get the public IP address of the host01 from instances under AWS and access the instance to check if Apache is installed.

In the first place, we will not be able to access to the web server. First, we need to add port 80 to the Security Group we created before (launch-wizard-1). Otherwise, HTTP access stays blocked.

Now can access and check if Apache is installed.

Step 3: Exploring the module service and shell

# Checking, Stopping and Starting the Apache service
ansible -i hosts host01 -m shell -a "systemctl status apache2"
ansible -i hosts host01 -m shell -a "systemctl stop apache2" -b

We cannot connect to the server anymore because we stopped.

# restart the Apache Service
ansible -i hosts host01 -m service -a "name=apache2 state=started" -b

Step 4: Exploring the copy and file module

# Create a dummy file inside ansible-tasks folder
echo "This is an example file" >> dump

Copy this dump file to all instances with Ansible.

#check if the files is already inside hosts, we do not want to overwrite anything
ansible -i hosts all -m command -a 'ls /tmp/dump'
#copy the file to all hosts
ansible -i hosts all -m copy -a "src=dump dest=/tmp"
#check again
ansible -i hosts all -m command -a 'ls /tmp/dump'
#print file content inside host01
ansible -i hosts host01 -m command -a "cat /tmp/dump"

# Delete the 'dump' file 
ansible -i hosts all -m file -a 'path=/tmp/dump state=absent'

Step 5: Destroy the resources

If you are doing only for learning practices do not forget to destroy the resources you created. Otherwise, AWS keeps you charging for running resources. Terminate host01 and host02 instances on the AWS management console.

CONGRATULATIONS!!!

Follow me on LinkedIn

https://www.linkedin.com/in/cansu-t-a5617557/

REFERENCES

The Cloud Bootcamp Platform - EN

Installing Ansible - Ansible Documentation

Ansible Configuration Settings - Ansible Documentation

ansible/examples/ansible.cfg at stable-2.9 · ansible/ansible

Using Ansible modules and plugins - Ansible Documentation

ansible.builtin.apt module - Manages apt-packages - Ansible Documentation

ansible.builtin.service module - Manage services - Ansible Documentation

ansible.builtin.copy module - Copy files to remote locations - Ansible Documentation

ansible.builtin.file module - Manage files and file properties - Ansible Documentation

ansible.builtin.command module - Execute commands on targets - Ansible Documentation