Part 13 — HumanGov Application — Ansible-Final Project: Configuration and Deployment of HumanGov SaaS Application on AWS EC2 Inventory Across US States Using Ansible and Securily Storing Configuration Files on AWS CodeCommit

Cansu Tekin
17 min readMar 18, 2024

--

The HumanGov Application is a Human Resources Management Cloud SaaS application for the Department of Education across all 50 states in the US. Whenever a state hired a new employee the registiration of the new employees will be done through the application. Their information will be stored inside AWS DynamoDB and their employment document will be stored inside S3 buckets. Check the first 10 parts where we focused on Terraform and then we focused on Ansible in the Part 11 and Part 12. Now, we are going to finalize Ansible section and keep improving the application architecture with DevOps tools in the following sections.

In this following project series, we are going to transition the architecture from a traditional virtual machine architecture to a modern container-based architecture using Docker containers and Kubernetes running on AWS. In addition, we will also be responsible for automating the complete software delivery process using Pipelines CI/CD using AWS services such as AWS CodeCommit, AWS CodePipeline, AWS CodeBuild, and AWS CodeDeploy. Finally, we will learn how to monitor and observe the cloud environment in real-time using tools such as Prometheus, Grafana, and automate one-off cloud tasks using Python and the AWS SDK.

Summary: We already have Terraform configuration codes to deploy AWS infrastructure and run the HumanGov python application for the 50 states from previous parts. In this final project, we will write an Ansible role and Ansible playbook to run the HumanGov application on top of this infrastructure. We will commit and push these Ansible files to the AWS CodeCommit to keep our files in a Git repository. Additionally, we need to update our Terraform files because the application running inside the EC2 instance needs to connect to DynamoDB to store data and the S3 bucket to upload necessary files coming through the application. We should authenticate the phyton application to do these operations by assigning necessary security roles to EC2 instances. After that, we are going to deploy the HumanGov application inside the EC2 instance in parallel across all tenants’ infrastructure in a fully automated way.

PART 1: Pre-requisites — Configuration of the Infrastructure with Terraform

PART 2: Deploy and Configure the HumanGov Application Manually

PART 3: Deploy the HumanGov application with Ansible

PART 1: Pre-requisites — Configuration of the Infrastructure with Terraform

Before starting heavily on the Ansible part we should have other infrastructure configurations ready for the HumanGov application.

When a new employee needs to be added to the system with the HumanGov Python application, the application will send a request to the NGINX web server running inside the EC2 instance. The NGINX is an intermediary between client devices (such as web browsers) and application servers that works as a reverse proxy allowing it to sit in front of application servers as a single entry point, forwards requests to the appropriate servers like the WSGI application server, in our case, and sends responses back to the clients. We will connect NGINX and the WSGI application server with the Unix socket in the configuration file. The application server will pass the request to the Python Flask web application. After that, the employee information will be written to the AWS DynomaDB table and the employee document will be uploaded to the S3 bucket by Python Flask Application.

Step 1: Configure Cloud 9 with non-temporary credentials

Cloud9 comes with temporary credentials that allow us to connect to the AWS services. We did not specify any keys for authentication while using Cloud9 so far because it comes with it. Cloud9 manages these access credentials for us. These credentials are obtained through IAM roles and have a limited duration.

The non-temporary credentials provide long-term access to AWS resources without time limitations. We need to create non-temporary credentials using the IAM module and disable the temporary credentials. That will allow us to create an IAM user with full privileges to configure the infrastructure to create the necessary architecture.

Create a new user on IAM with Admin privilege.

Go to AWS Services > IAM > Users > Create user > Next > attach policies directly > pick AdministratorAccess > Create User

Go to inside of the cloud9-user we just created. We will create security credentials for this user to authenticate and authorize requests made to AWS services.

Security credentials tab> Access keys > Create access key

Select the use case as Command Line Interface (CLI) because this user will be used by Terraform to create resources.

Disable the temporary credentials on Cloud9 :

Settings > AWS Settings > Credentials > Turning Off the option “AWS managed temporary credentials”

Now, we can configure Cloud9 to use the user cloud9-user we have created which has all the privileges.

Configure the new IAM user credentials by running the aws configure command.

It will ask for the access key and secret access key we created before. Go to the IAM Management Console, copy the keys, and paste them into the Cloud9 terminal. Cloud9 will use our user privileges now, which are AdministratorAccess privileges. It will allow Terraform to create the infrastructure we want.

Step 2: Update the HumanGov Terraform module

We should not have any infrastructure because we destroyed what we created before. Check if everything is fine with the terraform files. You can check all the details related to the Terraform configuration files we created in the previous part of the project series.

cd human-gov-infrastructure/terraform
terraform show
terraform validate

We first let Terraform create and attach the IAM role to the EC2 instances we will create. This will give privileges to EC2 instances to read data from DynamoDB and S3 bucket and write data to DynmoDB and S3 bucket. Add the IAM role code below after the last line of the file modules/aws_humangov_infrastructure/main.tf:

resource "aws_iam_role" "s3_dynamodb_full_access_role" {
name = "humangov-${var.state_name}-s3_dynamodb_full_access_role"

assume_role_policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Principal": {
"Service": "ec2.amazonaws.com"
},
"Effect": "Allow",
"Sid": ""
}
]
}
EOF

tags = {
Name = "humangov-${var.state_name}"
}

}

resource "aws_iam_role_policy_attachment" "s3_full_access_role_policy_attachment" {
role = aws_iam_role.s3_dynamodb_full_access_role.name
policy_arn = "arn:aws:iam::aws:policy/AmazonS3FullAccess"

}

resource "aws_iam_role_policy_attachment" "dynamodb_full_access_role_policy_attachment" {
role = aws_iam_role.s3_dynamodb_full_access_role.name
policy_arn = "arn:aws:iam::aws:policy/AmazonDynamoDBFullAccess"

}

resource "aws_iam_instance_profile" "s3_dynamodb_full_access_instance_profile" {
name = "humangov-${var.state_name}-s3_dynamodb_full_access_instance_profile"
role = aws_iam_role.s3_dynamodb_full_access_role.name

tags = {
Name = "humangov-${var.state_name}"
}
}

Add the argument below to the EC2 instance resource in the file: modules/aws_humangov_infrastructure/main.tf

iam_instance_profile = aws_iam_instance_profile.s3_dynamodb_full_access_instance_profile.name

The EC2 instance resource will look like this after the update:

resource "aws_instance" "state_ec2" {
ami = "ami-007855ac798b5175e"
instance_type = "t2.micro"
key_name = "humangov-ec2-key"
vpc_security_group_ids = [aws_security_group.state_ec2_sg.id]
iam_instance_profile = aws_iam_instance_profile.s3_dynamodb_full_access_instance_profile.name

tags = {
Name = "humangov-${var.state_name}"
}
}

Update the security groups as below in the file: modules/aws_humangov_infrastructure/main.tf

  ingress {
from_port = 80 #HTTP port, NGINX port to access to the HumanGov Application
to_port = 80
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}

ingress { # Port for Python Flask Application,optional, we may use for troubleshooting. Users will reach the application via NGINX port and pass the request to the Flask Application as we explained before.
from_port = 5000
to_port = 5000
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}

ingress { # This port for Cloud9 to make SSH connection to the EC2 instance with private IP address to run Ansible.
from_port = 0
to_port = 0
protocol = "-1"
security_groups = ["<YOUR_CLOUD9_SECGROUP>"]
}

egress { # This is the port allowing EC2 instance connecting to the internet and download packages required for our application like NGINX..
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}

Cloud9 Security Group:

Make sure you have a private key that will be used to make an SSH connection to the EC2 instance. We created it before with the name the humangov-ec2-key. Go to AWS EC2 instances > Network & Security > Key pairs > humangov-ec2-key. If you do not have create a new one with the same name and download it to your computer. If you have downloaded it to your computer before, upload it to the humangov folder inside the Cloud9 environment.

Step 3: Provision the infrastructure on AWS using Terraform

Go to the environment/human-gov-infrastructure/terraform/

terraform plan
terraform apply

Terraform should create 9 resources per state. We have only one state in our state list in variables.tf file we created the previous project series.

Step 4: Commit the changes to the local Git repository, CodeCommit

You can read more here about CodeCommit.

git status
git add .
git status
git commit -m "Added IAM Role to Terraform module aws_humangov_infrastructure/main.tf"

Push the changes to the CodeCommit repository.

git push -u origin 

This command will give an error because we disabled managed credentials on Cloud9. It will also affect how Cloud9 authenticates CodeCommit. We will handle this error at the end of the project while pushing all the changes once we are done!

PART 2: Deploy and Configure the HumanGov Application on the AWS EC2 instance Manually

Step 1: Download the Humangov application files

First, we need to get Humangov application files. Go to the human-gov-application git folder that we created before to keep application files. It was empty so far. Download the files from the given directory. Create an src directory to store application files in it.

cd ~/environment/human-gov-application
echo "*.zip" >> .gitignore
mkdir src
cd src
wget https://tcb-bootcamps.s3.amazonaws.com/tcb5001-devopscloud-bootcamp/v2/module4-ansible/humangov-app.zip
unzip humangov-app.zip

Step 2: Commit the recent changes in the Cloud9 local Git repository

git status
git add .
git add ../.gitignore
git commit -m "HumanGov 1st commit"

Step 3: Connect to the EC2 instance and configure to deploy the HumanGov Application

# Connect to the EC2 instance with SSH 
chmod 600 /home/ec2-user/environment/humangov-ec2-key.cer
ssh -i "humangov-ec2-key.cer" ubuntu@ec2-54-205-101-211.compute-1.amazonaws.com

# Update and upgrade packages
sudo apt-get update
sudo apt-get upgrade -y

# Install required packages for the app
sudo apt-get install -y nginx python3-pip python3-dev build-essential libssl-dev libffi-dev python3-setuptools python3-venv unzip

# UFW (Uncomplicated Firewall) is a front-end for iptables on Linux systems.
# It allowsspecific types of traffic that can reach the desired services on your server.
# By default, UFW denies all incoming connections and allows all outgoing connections.
# Allow Nginx HTTP traffic through the UFW (Uncomplicated Firewall)
sudo ufw allow 'Nginx HTTP'
# Set necessary environment variables
export project_path=/home/ubuntu/humangov
export username=ubuntu
export project_name=humangov

# Create a project directory inside EC2 instance using environment variables you created
mkdir -p $project_path

# Change the owner of the directory specified by $project_path to the user specified by the $username variable
sudo chown $username:$username $project_path

# Sets the permissions to rwxr-xr-x, meaning the owner has read, write, and execute permissions, while others have only read and execute permissions.
sudo chmod 0755 $project_path

# Create Python virtual environment within the directory specified by the $project_path variable
# This virtual environment is isolated from the system's Python installation,
# Allows you to install and manage Python packages specific to your project without affecting other projects or the system-wide Python installation
python3 -m venv $project_path/humangovenv
# Copy the application zip file to the EC2 instance by exiting or opening a new terminal
scp -i /home/ec2-user/environment/humangov-ec2-key.cer /home/ec2-user/environment/human-gov-application/src/humangov-app.zip ubuntu@ip-172-31-30-181:/home/ubuntu/humangov

# Unzip the application zip file, extract the contents of the ZIP file to the project path
unzip $project_path/humangov-app.zip -d $project_path
# Install Python packages from requirements.txt into the virtual environment
$project_path/humangovenv/bin/pip install -r $project_path/requirements.txt

Create a systemd service file for Gunicorn (replace values with yours in the template):

A systemd service file for Gunicorn is a configuration file that defines how Gunicorn, a popular WSGI HTTP server for Python web applications like Django or Flask, should be managed by the systemd init system on Linux systems.

sudo tee /etc/systemd/system/$project_name.service <<EOL
[Unit]
Description=Gunicorn instance to serve $project_name
After=network.target

[Service]
User=$username
Group=www-data
WorkingDirectory=$project_path
Environment="PATH=$project_path/humangovenv/bin"
Environment="AWS_REGION=us-east-1"
Environment="AWS_DYNAMODB_TABLE=humangov-california-dynamodb"
Environment="AWS_BUCKET=humangov-california-s3-pl32"
Environment="US_STATE=california"
ExecStart=$project_path/humangovenv/bin/gunicorn --workers 1 --bind unix:$project_path/$project_name.sock -m 007 $project_name:app

[Install]
WantedBy=multi-user.target
EOL
# Change permissions of the user’s home directory
sudo chmod 0755 /home/$username

# Remove the default nginx configuration file
sudo rm /etc/nginx/sites-enabled/default

Configure Nginx to proxy requests. The content will be written to the NGINX configuration file.

sudo tee /etc/nginx/sites-available/$project_name <<EOL
server {
listen 80;
server_name $project_name www.$project_name;
location / {
include proxy_params;
proxy_pass http://unix:$project_path/$project_name.sock;
}
}
EOL
# Enable and start Gunicorn service
sudo systemctl enable $project_name
sudo systemctl start $project_name
sudo systemctl status $project_name

# press q to return the main bash prompt
# Enable Nginx configuration, creates a symbolic link,it becomes easier to manage configurations, especially in environments with multiple NGINX sites. 
# NGINX typically reads its configuration files from the "/etc/nginx/sites-enabled/" directory.
# /etc/nginx/sites-available/$project_name: This is the path to the NGINX site configuration file that you want to enable. $project_name is a placeholder for the name of your specific site configuration file.
# /etc/nginx/sites-enabled/: This is the directory where enabled site configurations are stored. By creating a symbolic link here, NGINX will include the configuration from the source file in its active configuration.
sudo ln -s /etc/nginx/sites-available/$project_name /etc/nginx/sites-enabled/

# Restart Nginx and the humangov service
sudo systemctl restart $project_name
sudo systemctl status $project_name
sudo systemctl restart nginx
sudo systemctl status nginx

# pres q to return the main bash prompt

Step 5: Test the HumanGov Application

Go to the EC2 instance public IP address to connect to the application.

It is working!

Step 6: Destroy the infrastructure using Terraform

cd ~/environment/human-gov-infrastructure/terraform
terraform destroy

PART 3: Deploy the HumanGov application with Ansible

Terraform manages infrastructure provisioning, while Ansible is responsible for configuring and deploying applications on those resources.

Step 1: Update Terraform files for infrastructure provisioning

We keep using Terraform for infrastructure provisioning with some updates.

If Ansible is installed in your environment you should already have a default host inventory file under /etc/ansible/hosts.

In the Cloud9 environment, we have Ansible pre-installed already. Otherwise, make sure it is installed:

# Verify whether Ansible is installed in your Cloud9 environment
ansible --version

# Otherwise, install it
sudo yum install epel-release
sudo yum install ansible

Create the empty Ansible inventory file on /etc/ansible/hosts

sudo touch /etc/ansible/hosts
# Change the ownership of the "hosts" file to the user ec2-user to ensures that the ec2-user has the necessary permissions to modify the hosts file.
sudo chown ec2-user:ec2-user /etc/ansible/hosts
# Recursively change the ownership of the entire "/etc/ansible" directory and its contents to ensure that the ec2-user has full control over the Ansible configuration directory.
sudo chown -R ec2-user:ec2-user /etc/ansible

Update the module variable file, modules/aws_humangov_infrastructure/variables.tf, by adding the region variable to the

variable "state_name" {
description = "The name of the US State"
}
variable "region" {
default = "us-east-1"
}

Add provisioners to EC2 resource in modules/aws_humangov_infrastructure/main.tf. The provisioners will execute commands on the local machine during instance creation.

resource "aws_instance" "state_ec2" {
ami = "ami-007855ac798b5175e"
instance_type = "t2.micro"
key_name = "humangov-ec2-key"
vpc_security_group_ids = [aws_security_group.state_ec2_sg.id]
iam_instance_profile = aws_iam_instance_profile.s3_dynamodb_full_access_instance_profile.name

# Add the EC2 instance's private IP address to the local machine's known_hosts file after waiting for 30 seconds.
provisioner "local-exec" {
command = "sleep 30; ssh-keyscan ${self.private_ip} >> ~/.ssh/known_hosts"
}

# Append EC2 instance information to the /etc/ansible/hosts file, enabling Ansible to manage the instance.
provisioner "local-exec" {
command = "echo ${var.state_name} id=${self.id} ansible_host=${self.private_ip} ansible_user=ubuntu us_state=${var.state_name} aws_region=${var.region} aws_s3_bucket=${aws_s3_bucket.state_s3.bucket} aws_dynamodb_table=${aws_dynamodb_table.state_dynamodb.name} >> /etc/ansible/hosts"
}

# Remove the information about the EC2 instance from the /etc/ansible/hosts file when the instance is destroyed.
provisioner "local-exec" {
provisioner "local-exec" {
command = "sed -i '/${self.id}/d' /etc/ansible/hosts"
when = destroy
}

tags = {
Name = "humangov-${var.state_name}"
}
}

Provision the infrastructure with updated Terraform files. I am going to do it for only two states for now. I updated the human-gov-infrastructure/terraform/variable.tf file

variable "states" {
description = "A list of state names"
default = ["california", "texas"]
}
terraform fmt
terraform validate
terraform plan
terraform apply

Step 3: Commit and push changes to the Cloud9 local git repository

git status
git add .
git status
git commit -m "Terraform files updated"

Step 4: Ping the instances

Let’s check if our instances for California and Texas are up and running.

ansible all -m ping -e "ansible_ssh_private_key_file=/home/ec2-user/environment/humangov-ec2-key.cer"

Step 5: Create an Ansible role for the HumanGov Application

First, create a folder to store Ansible files, then create the role files.

cd ~/environment/human-gov-infrastructure 
mkdir ansible
mkdir roles
ansible-galaxy init roles/humangov_webapp

This command will create a directory with the specified role name and generate the standard role structure inside it including their main.yml files.

Next, we can add the additional files we will use in this role.

cd ansible
touch roles/humangov_webapp/templates/nginx.conf.j2
touch roles/humangov_webapp/templates/humangov.service.j2
touch deploy-humangov.yml

Step 6: Define the Ansible role under roles/humangov_webapp role with tasks and other configurations to deploy the HumanGov web application

defaults/main.yml

---
username: ubuntu
project_name: humangov
project_path: "/home/{{ username }}/{{ project_name }}"
source_application_path: /home/ec2-user/environment/human-gov-application/src

handlers/main.yml

---
- name: Restart Nginx
systemd:
name: nginx
state: restarted
become: yes
- name: Restart humangov
systemd:
name: humangov
state: restarted
become: yes

tasks/main.yml

---
- name: Update and upgrade apt packages
apt:
upgrade: dist
update_cache: yes
become: yes
- name: Install required packages
apt:
name:
- nginx
- python3-pip
- python3-dev
- build-essential
- libssl-dev
- libffi-dev
- python3-setuptools
- python3-venv
- unzip
state: present
become: yes
- name: Ensure UFW allows Nginx HTTP traffic
ufw:
rule: allow
name: 'Nginx HTTP'
become: yes
- name: Create project directory
file:
path: "{{ project_path }}"
state: directory
owner: "{{ username }}"
group: "{{ username }}"
mode: '0755'
become: yes
- name: Create Python virtual environment
command:
cmd: python3 -m venv {{ project_path }}/humangovenv
creates: "{{ project_path }}/humangovenv"
- name: Copy the application zip file to the destination
copy:
src: "{{ source_application_path }}/humangov-app.zip"
dest: "{{ project_path }}"
owner: "{{ username }}"
group: "{{ username }}"
mode: '0644'
become: yes
- name: Unzip the application zip file
unarchive:
src: "{{ project_path }}/humangov-app.zip"
dest: "{{ project_path }}"
remote_src: yes
notify: Restart humangov
become: yes
- name: Install Python packages from requirements.txt into the virtual environment
pip:
requirements: "{{ project_path }}/requirements.txt"
virtualenv: "{{ project_path }}/humangovenv"
- name: Create systemd service file for Gunicorn
template:
src: humangov.service.j2
dest: /etc/systemd/system/{{ project_name }}.service
notify: Restart humangov
become: yes
- name: Enable and start Gunicorn service
systemd:
name: "{{ project_name }}"
enabled: yes
state: started
become: yes
- name: Remove the default nginx configuration file
file:
path: /etc/nginx/sites-enabled/default
state: absent
become: yes
- name: Change permissions of the user's home directory
file:
path: "/home/{{ username }}"
mode: '0755'
become: yes
- name: Configure Nginx to proxy requests
template:
src: nginx.conf.j2
dest: /etc/nginx/sites-available/{{ project_name }}
become: yes
- name: Enable Nginx configuration
file:
src: /etc/nginx/sites-available/{{ project_name }}
dest: /etc/nginx/sites-enabled/{{ project_name }}
state: link
notify: Restart Nginx
become: yes

templates/humangov.service.j2: Jinja2 template for the Gunicorn systemd service.

[Unit]
Description=Gunicorn instance to serve {{ project_name }}
After=network.target
[Service]
User={{ username }}
Group=www-data
WorkingDirectory={{ project_path }}
Environment="US_STATE={{ us_state }}"
Environment="PATH={{ project_path }}/humangovenv/bin"
Environment="AWS_REGION={{ aws_region }}"
Environment="AWS_DYNAMODB_TABLE={{ aws_dynamodb_table }}"
Environment="AWS_BUCKET={{ aws_s3_bucket }}"
ExecStart={{ project_path }}/humangovenv/bin/gunicorn --workers 1 --bind unix:{{ project_path }}/{{ project_name }}.sock -m 007 {{ project_name }}:app
[Install]
WantedBy=multi-user.target

templates/nginx.conf.j2: Jinja2 template for the Nginx configuration

server {
listen 80;
server_name humangov www.humangov;

location / {
include proxy_params;
proxy_pass http://unix:{{ project_path }}/{{ project_name }}.sock;
}
}

deploy-humangov.yml

- hosts: all
roles:
- humangov_webapp

Step 7: Run the Ansible playbook to deploy the HumanGov Web Application

ansible-playbook deploy-humangov.yml -e "ansible_ssh_private_key_file=/home/ec2-user/environment/humangov-ec2-key.cer"

Step 8:Test the HumanGov Application by opening up the Public DNS for both state

Outputs:

state_infrastructure_outputs = {
"california" = {
"dynamodb_table" = "humangov-california-dynamodb"
"ec2_public_dns" = "ec2-52-72-238-79.compute-1.amazonaws.com"
"s3_bucket" = "humangov-california-s3-tw1b"
}
"texas" = {
"dynamodb_table" = "humangov-texas-dynamodb"
"ec2_public_dns" = "ec2-54-174-73-108.compute-1.amazonaws.com"
"s3_bucket" = "humangov-texas-s3-7v5k"
}
}

Add a new employee:

Check if you are able to write employee data to DynamoDB table and store employe documents in S3 bucket:

Step 9: Commit and push the application source codes with the latest changes to the CodeCommit Git repository

Enable back the temporary credentials on Cloud9

Settings > AWS Settings > Credentials > Turning Off the option “AWS managed temporary credentials”

# Commit and push changes to AWS Code Commit
cd ~/environment/human-gov-infrastructure
git status
git add .
git status
git commit -m "Ansible configuration 1st commit for California and Texas"

git push -u origin
# Push the application source code to the AWS CodeCommit remote repository
cd ~/environment/human-gov-application
git pull --rebase
git push -u origin

Step 10: Destroy the infrastructure with Terraform

First, fix the Ansible directory permission

sudo chown -R ec2-user:ec2-user /etc/ansible
export AWS_ACCESS_KEY_ID="AKIAZDSAVGW7LGTR6LNZ"
export AWS_SECRET_ACCESS_KEY="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"

cd ~/environment/human-gov-infrastructure/terraform
terraform destroy

CONGRATULATIONS!!!

--

--

Cansu Tekin

AWS Community Builder | Full Stack Java Developer | DevOps | AWS | Microsoft Azure | Google Cloud | Docker | Kubernetes | Ansible | Terraform