HumanGov: Configuration and Deployment of HumanGov SaaS Application on AWS EC2 Inventory Across US States Using Ansible Securely Storing Configuration Files on AWS Code Commit

Sampath P
12 min readSep 2, 2023

--

In this real-life project, I used Ansible, a tool for automating configurations, to set up and launch the HumanGov SaaS application on AWS EC2 servers. These servers are going to be used to provide the HumanGov service to people all across the United States.

Hands-on Project: HumanGov | Ansible + AWS Code Commit — Implementation — Part 1 | AWS

Overview of the application architecture

Part 1

Pre-requisites

Configure Cloud 9 with non-temporary credentials

  • Create a new user on IAM with Admin privilege
cloud9 user
  • Disable the temporary credentials on Cloud9 by clicking on Settings > AWS Settings > Credentials > Turning Off the option “AWS managed temporary credentials”
temporary credentials disabled
  • Configure the new IAM user credentials by running the aws configure command
aws configure
  • Update the HumanGov Terraform module as below:
cd human-gov-infrastructure/terraform
terraform show
terraform validate
terraform plan
terraform apply
terraform destroy -auto-approve

Add the IAM role code below after the last line of the file modules/aws_humangov_infrastructure/main.tf:

resource "aws_iam_role" "s3_dynamodb_full_access_role" {
name = "humangov-${var.state_name}-s3_dynamodb_full_access_role"

assume_role_policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Principal": {
"Service": "ec2.amazonaws.com"
},
"Effect": "Allow",
"Sid": ""
}
]
}
EOF

tags = {
Name = "humangov-${var.state_name}"
}

}

resource "aws_iam_role_policy_attachment" "s3_full_access_role_policy_attachment" {
role = aws_iam_role.s3_dynamodb_full_access_role.name
policy_arn = "arn:aws:iam::aws:policy/AmazonS3FullAccess"

}

resource "aws_iam_role_policy_attachment" "dynamodb_full_access_role_policy_attachment" {
role = aws_iam_role.s3_dynamodb_full_access_role.name
policy_arn = "arn:aws:iam::aws:policy/AmazonDynamoDBFullAccess"

}

resource "aws_iam_instance_profile" "s3_dynamodb_full_access_instance_profile" {
name = "humangov-${var.state_name}-s3_dynamodb_full_access_instance_profile"
role = aws_iam_role.s3_dynamodb_full_access_role.name

tags = {
Name = "humangov-${var.state_name}"
}
}

Add the argument below to the EC2 instance resource in the file: modules/aws_humangov_infrastructure/main.tf

iam_instance_profile = aws_iam_instance_profile.s3_dynamodb_full_access_instance_profile.name

Update the security groups as below in the file: modules/aws_humangov_infrastructure/main.tf


ingress {
from_port = 80
to_port = 80
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}

ingress {
from_port = 5000
to_port = 5000
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}

ingress {
from_port = 0
to_port = 0
protocol = "-1"
security_groups = ["<YOUR_CLOUD9_SECGROUP>"]
}

egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}

💡 PS: If you’re prefer to replace the whole file main.tf file with the new code, use the code inside of this section below

resource "aws_security_group" "state_ec2_sg" {
name = "humangov-${var.state_name}-ec2-sg"
description = "Allow traffic on ports 22 and 80"

ingress {
from_port = 80
to_port = 80
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}

ingress {
from_port = 5000
to_port = 5000
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}

ingress {
from_port = 0
to_port = 0
protocol = "-1"
security_groups = ["<YOUR_CLOUD9_SECGROUP_ID"]
}

egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}

tags = {
Name = "humangov-${var.state_name}"
}
}

resource "aws_instance" "state_ec2" {
ami = "ami-007855ac798b5175e"
instance_type = "t2.micro"
key_name = "humangov-ec2-key"
vpc_security_group_ids = [aws_security_group.state_ec2_sg.id]
iam_instance_profile = aws_iam_instance_profile.s3_dynamodb_full_access_instance_profile.name

tags = {
Name = "humangov-${var.state_name}"
}
}

resource "aws_dynamodb_table" "state_dynamodb" {
name = "humangov-${var.state_name}-dynamodb"
billing_mode = "PAY_PER_REQUEST"
hash_key = "id"

attribute {
name = "id"
type = "S"
}

tags = {
Name = "humangov-${var.state_name}"
}
}

resource "random_string" "bucket_suffix" {
length = 4
special = false
upper = false
}

resource "aws_s3_bucket" "state_s3" {
bucket = "humangov-${var.state_name}-s3-${random_string.bucket_suffix.result}"

tags = {
Name = "humangov-${var.state_name}"
}
}

resource "aws_iam_role" "s3_dynamodb_full_access_role" {
name = "humangov-${var.state_name}-s3_dynamodb_full_access_role"

assume_role_policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Principal": {
"Service": "ec2.amazonaws.com"
},
"Effect": "Allow",
"Sid": ""
}
]
}
EOF

tags = {
Name = "humangov-${var.state_name}"
}

}

resource "aws_iam_role_policy_attachment" "s3_full_access_role_policy_attachment" {
role = aws_iam_role.s3_dynamodb_full_access_role.name
policy_arn = "arn:aws:iam::aws:policy/AmazonS3FullAccess"

}

resource "aws_iam_role_policy_attachment" "dynamodb_full_access_role_policy_attachment" {
role = aws_iam_role.s3_dynamodb_full_access_role.name
policy_arn = "arn:aws:iam::aws:policy/AmazonDynamoDBFullAccess"

}

resource "aws_iam_instance_profile" "s3_dynamodb_full_access_instance_profile" {
name = "humangov-${var.state_name}-s3_dynamodb_full_access_instance_profile"
role = aws_iam_role.s3_dynamodb_full_access_role.name

tags = {
Name = "humangov-${var.state_name}"
}
}

💡 Delete and Create a new SSH Key Pair on EC2 Dashboard with the name humangov-ec2-key to match with the key-name inside of the Terraform module and upload it to /home/ec2-user/environment/ on AWS Cloud9

Provision the infrastructure on AWS using Terraform

terraform plan
terraform apply
  • Commit the changes to the local Git repository
git status
git add .
git status
git commit -m "Added IAM Role to Terraform module aws_humangov_infrastructure/main.tf"
committed changes to git repository

Note: The git push command doesn’t work because if you remember that we switched off the temporary credentials of cloud9 and now we are using a different user. Hence it will not go through.

Part 2

Deploying the app without Ansible

Download the HumanGov application to the local Git Repository on AWS Cloud9

cd ~/environment/human-gov-application
echo "*.zip" >> .gitignore
mkdir src
cd src
wget <https://tcb-bootcamps.s3.amazonaws.com/tcb5001-devopscloud-bootcamp/v2/module4-ansible/humangov-app.zip>
unzip humangov-app.zip
Output

Push the application source to the local Git repository

git status
git add .
git add ../.gitignore
git commit -m "HumanGov app 1st commit"

Installing and Configuring the app manually

Connect to the EC2 instance

chmod 600 /home/ec2-user/environment/humangov-ec2-key.pem 
ssh -i /home/ec2-user/environment/humangov-ec2-key.pem ubuntu@<PRIVATE_IP>

Update and upgrade apt packages:

sudo apt-get update
sudo apt-get upgrade -y

Install required packages:

sudo apt-get install -y nginx python3-pip python3-dev build-essential libssl-dev libffi-dev python3-setuptools python3-venv unzip

Ensure UFW allows Nginx HTTP traffic:

sudo ufw allow 'Nginx HTTP'

Create project directory:

export project_path=/home/ubuntu/humangov
export username=ubuntu

mkdir -p $project_path
sudo chown $username:$username $project_path
sudo chmod 0755 $project_path

Create Python virtual environment:

python3 -m venv $project_path/humangovenv

Copy the application zip file to the destination:

exit

scp -i /home/ec2-user/environment/humangov-ec2-key.pem humangov-app.zip ubuntu@<PRIVATE_IP>:/home/ubuntu/humangov

ssh -i /home/ec2-user/environment/humangov-ec2-key.pem ubuntu@<PRIVATE_IP>

Unzip the application zip file:

export project_path=/home/ubuntu/humangov
export username=ubuntu
export project_name=humangov

unzip $project_path/humangov-app.zip -d $project_path

Install Python packages from requirements.txt into the virtual environment:

$project_path/humangovenv/bin/pip install -r $project_path/requirements.txt

Create systemd service file for Gunicorn (values need to be replaced in the template):

sudo tee /etc/systemd/system/$project_name.service <<EOL
[Unit]
Description=Gunicorn instance to serve $project_name
After=network.target

[Service]
User=$username
Group=www-data
WorkingDirectory=$project_path
Environment="PATH=$project_path/humangovenv/bin"
**Environment="AWS_REGION=us-east-1"
Environment="AWS_DYNAMODB_TABLE=humangov-california-dynamodb"
Environment="AWS_BUCKET=humangov-california-s3-ku6m"
Environment="US_STATE=california"**
ExecStart=$project_path/humangovenv/bin/gunicorn --workers 1 --bind unix:$project_path/$project_name.sock -m 007 $project_name:app

[Install]
WantedBy=multi-user.target
EOL
creating a service file

Change permissions of the user’s home directory:

sudo chmod 0755 /home/$username

Remove the default nginx configuration file:

sudo rm /etc/nginx/sites-enabled/default

Configure Nginx to proxy requests (values need to be replaced in the template):

sudo tee /etc/nginx/sites-available/$project_name <<EOL
server {
listen 80;
server_name $project_name www.$project_name;
location / {
include proxy_params;
proxy_pass <http://unix>:$project_path/$project_name.sock;
}
}
EOL

Enable and start Gunicorn service:

sudo systemctl enable $project_name

sudo systemctl start $project_name

sudo systemctl status $project_name

Enable Nginx configuration:

sudo ln -s /etc/nginx/sites-available/$project_name /etc/nginx/sites-enabled/

Restart Nginx and the humangov service:

sudo systemctl restart $project_name

sudo systemctl status $project_name

sudo systemctl restart nginx

sudo systemctl status nginx

Test the app

PDF

Successfully deployed the application

Destroy the infrastructure on AWS using Terraform

cd ~/environment/human-gov-infrastructure/terraform
terraform destroy

Part 3

Provisioning the AWS infrastructure with Terraform

Add the provisioners to EC2 resource in Terraform module file modules/aws_humangov_infrastructure/main.tf

 provisioner "local-exec" {
command = "sleep 30; ssh-keyscan ${self.private_ip} >> ~/.ssh/known_hosts"
}

provisioner "local-exec" {
command = "echo ${var.state_name} id=${self.id} ansible_host=${self.private_ip} ansible_user=ubuntu us_state=${var.state_name} aws_region=${var.region} aws_s3_bucket=${aws_s3_bucket.state_s3.bucket} aws_dynamodb_table=${aws_dynamodb_table.state_dynamodb.name} >> /etc/ansible/hosts"
}

provisioner "local-exec" {
command = "sed -i '/${self.id}/d' /etc/ansible/hosts"
when = destroy
}

💡 PS: If you’re or prefer to replace the whole file main.tf file with the new code, use the code inside of this section below

resource "aws_security_group" "state_ec2_sg" {
name = "humangov-${var.state_name}-ec2-sg"
description = "Allow traffic on ports 22 and 80"

ingress {
from_port = 80
to_port = 80
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}

ingress {
from_port = 5000
to_port = 5000
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}

ingress {
from_port = 0
to_port = 0
protocol = "-1"
security_groups = ["sg-027f57abd3fefda49"]
}

egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}

tags = {
Name = "humangov-${var.state_name}"
}
}
resource "aws_instance" "state_ec2" {
ami = "ami-007855ac798b5175e"
instance_type = "t2.micro"
key_name = "humangov-ec2-key"
vpc_security_group_ids = [aws_security_group.state_ec2_sg.id]
iam_instance_profile = aws_iam_instance_profile.s3_dynamodb_full_access_instance_profile.name

provisioner "local-exec" {
command = "sleep 30; ssh-keyscan ${self.private_ip} >> ~/.ssh/known_hosts"
}

provisioner "local-exec" {
command = "echo ${var.state_name} id=${self.id} ansible_host=${self.private_ip} ansible_user=ubuntu us_state=${var.state_name} aws_region=${var.region} aws_s3_bucket=${aws_s3_bucket.state_s3.bucket} aws_dynamodb_table=${aws_dynamodb_table.state_dynamodb.name} >> /etc/ansible/hosts"
}

provisioner "local-exec" {
command = "sed -i '/${self.id}/d' /etc/ansible/hosts"
when = destroy
}

tags = {
Name = "humangov-${var.state_name}"
}
}
resource "aws_dynamodb_table" "state_dynamodb" {
name = "humangov-${var.state_name}-dynamodb"
billing_mode = "PAY_PER_REQUEST"
hash_key = "id"

attribute {
name = "id"
type = "S"
}

tags = {
Name = "humangov-${var.state_name}"
}
}
resource "random_string" "bucket_suffix" {
length = 4
special = false
upper = false
}
resource "aws_s3_bucket" "state_s3" {
bucket = "humangov-${var.state_name}-s3-${random_string.bucket_suffix.result}"

tags = {
Name = "humangov-${var.state_name}"
}
}

resource "aws_iam_role" "s3_dynamodb_full_access_role" {
name = "humangov-${var.state_name}-s3_dynamodb_full_access_role"
assume_role_policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Principal": {
"Service": "ec2.amazonaws.com"
},
"Effect": "Allow",
"Sid": ""
}
]
}
EOF
tags = {
Name = "humangov-${var.state_name}"
}

}

resource "aws_iam_role_policy_attachment" "s3_full_access_role_policy_attachment" {
role = aws_iam_role.s3_dynamodb_full_access_role.name
policy_arn = "arn:aws:iam::aws:policy/AmazonS3FullAccess"

}

resource "aws_iam_role_policy_attachment" "dynamodb_full_access_role_policy_attachment" {
role = aws_iam_role.s3_dynamodb_full_access_role.name
policy_arn = "arn:aws:iam::aws:policy/AmazonDynamoDBFullAccess"
}

resource "aws_iam_instance_profile" "s3_dynamodb_full_access_instance_profile" {
name = "humangov-${var.state_name}-s3_dynamodb_full_access_instance_profile"
role = aws_iam_role.s3_dynamodb_full_access_role.name
tags = {
Name = "humangov-${var.state_name}"
}
}

Add the variable region to the Terraform module variables file modules/aws_humangov_infrastructure/variables.tf

variable "state_name" {
description = "The name of the US State"
}

**variable "region" {
default = "us-east-1"
}**
Create the empty Ansible inventory file on /etc/ansible/hosts
su
sudo touch /etc/ansible/hosts
sudo chown ec2-user:ec2-user /etc/ansible/hosts
sudo chown -R ec2-user:ec2-user /etc/ansible

Provision the infrastructure

export 
terraform plan
terraform apply

Commit and push changes to the local Git repository

git status
git add .
git status
git commit -m "Added variable and provisioners to Terraform module aws_humangov_infrastructure/main.tf"

Deploying the app with Ansible

Below is an Ansible role to install and configure Nginx with a Flask application running under Gunicorn. This role assumes that you have a target system running Ubuntu and Ansible installed on a control machine.

Let’s start by creating a structure for our Ansible role, let’s call it humangov_webapp:

humangov_webapp/
├── defaults
│ └── main.yml
├── handlers
│ └── main.yml
├── tasks
│ └── main.yml
├── templates
│ ├── humangov.service.j2
│ └── nginx.conf.j2
└── vars
└── main.yml

Create the directory structure for an Ansible role using the following Linux commands.

cd ~/environment/human-gov-infrastructure 
mkdir ansible
cd ansible
mkdir -p roles/humangov_webapp/tasks
mkdir -p roles/humangov_webapp/handlers
mkdir -p roles/humangov_webapp/templates
mkdir -p roles/humangov_webapp/defaults
mkdir -p roles/humangov_webapp/vars
mkdir -p roles/humangov_webapp/files

Create the Ansible config file (ansible.cfg) with the content below

[defaults]
deprecation_warnings = False

Ping the instances created

ansible all -m ping -e "ansible_ssh_private_key_file=/home/ec2-user/environment/humangov-ec2-key.pem"

And you can create the necessary files using the touch command:

touch roles/humangov_webapp/tasks/main.yml
touch roles/humangov_webapp/handlers/main.yml
touch roles/humangov_webapp/templates/nginx.conf.j2
touch roles/humangov_webapp/templates/humangov.service.j2
touch roles/humangov_webapp/defaults/main.yml
touch roles/humangov_webapp/vars/main.yml
touch deploy-humangov.yml

defaults/main.yml

This file contains default values for our role variables:

---
username: ubuntu
project_name: humangov
project_path: "/home/{{ username }}/{{ project_name }}"
source_application_path: /home/ec2-user/environment/human-gov-application/src

handlers/main.yml

This file contains handlers that are triggered by tasks:

---
- name: Restart Nginx
systemd:
name: nginx
state: restarted
become: yes
- name: Restart humangov
systemd:
name: humangov
state: restarted
become: yes

tasks/main.yml

This is the main tasks file where all the tasks are defined:

---
- name: Update and upgrade apt packages
apt:
upgrade: dist
update_cache: yes
become: yes
- name: Install required packages
apt:
name:
- nginx
- python3-pip
- python3-dev
- build-essential
- libssl-dev
- libffi-dev
- python3-setuptools
- python3-venv
- unzip
state: present
become: yes
- name: Ensure UFW allows Nginx HTTP traffic
ufw:
rule: allow
name: 'Nginx HTTP'
become: yes
- name: Create project directory
file:
path: "{{ project_path }}"
state: directory
owner: "{{ username }}"
group: "{{ username }}"
mode: '0755'
become: yes
- name: Create Python virtual environment
command:
cmd: python3 -m venv {{ project_path }}/humangovenv
creates: "{{ project_path }}/humangovenv"
- name: Copy the application zip file to the destination
copy:
src: "{{ source_application_path }}/humangov-app.zip"
dest: "{{ project_path }}"
owner: "{{ username }}"
group: "{{ username }}"
mode: '0644'
become: yes

- name: Unzip the application zip file
unarchive:
src: "{{ project_path }}/humangov-app.zip"
dest: "{{ project_path }}"
remote_src: yes
notify: Restart humangov
become: yes
- name: Install Python packages from requirements.txt into the virtual environment
pip:
requirements: "{{ project_path }}/requirements.txt"
virtualenv: "{{ project_path }}/humangovenv"
- name: Create systemd service file for Gunicorn
template:
src: humangov.service.j2
dest: /etc/systemd/system/{{ project_name }}.service
notify: Restart humangov
become: yes
- name: Enable and start Gunicorn service
systemd:
name: "{{ project_name }}"
enabled: yes
state: started
become: yes
- name: Remove the default nginx configuration file
file:
path: /etc/nginx/sites-enabled/default
state: absent
become: yes
- name: Change permissions of the user's home directory
file:
path: "/home/{{ username }}"
mode: '0755'
become: yes
- name: Configure Nginx to proxy requests
template:
src: nginx.conf.j2
dest: /etc/nginx/sites-available/{{ project_name }}
become: yes
- name: Enable Nginx configuration
file:
src: /etc/nginx/sites-available/{{ project_name }}
dest: /etc/nginx/sites-enabled/{{ project_name }}
state: link
notify: Restart Nginx
become: yes

templates/humangov.service.j2

This is a Jinja2 template for the Gunicorn systemd service:

[Unit]
Description=Gunicorn instance to serve {{ project_name }}
After=network.target
[Service]
User={{ username }}
Group=www-data
WorkingDirectory={{ project_path }}
Environment="US_STATE={{ us_state }}"
Environment="PATH={{ project_path }}/humangovenv/bin"
Environment="AWS_REGION={{ aws_region }}"
Environment="AWS_DYNAMODB_TABLE={{ aws_dynamodb_table }}"
Environment="AWS_BUCKET={{ aws_s3_bucket }}"
ExecStart={{ project_path }}/humangovenv/bin/gunicorn --workers 1 --bind unix:{{ project_path }}/{{ project_name }}.sock -m 007 {{ project_name }}:app
[Install]
WantedBy=multi-user.target

templates/nginx.conf.j2

This is a Jinja2 template for the Nginx configuration:

server {
listen 80;
server_name humangov www.humangov;

location / {
include proxy_params;
proxy_pass <http://unix>:{{ project_path }}/{{ project_name }}.sock;
}
}

To use this Ansible role, update the playbook deploy-humangov.yml and include it in your playbook:

- hosts: all
roles:
- humangov_webapp

Run the Ansible Playbook command

ansible-playbook deploy-humangov.yml -e "ansible_ssh_private_key_file=/home/ec2-user/environment/humangov-ec2-key.pem"
ansible installing the application

Test the HumanGov app by opening up the Public DNS name

Application up and running

Add more states to Terraform /home/ec2-user/environment/human-gov-infrastructure/terraform/variables.tf

variable "states" {
description = "The list of state names"
default = ["california","florida","nevada"]
}

Provision of the new state infrastructure using Terraform

terraform apply
all Ec2 instances created
All hosts created

Re-run the playbook

ansible-playbook deploy-humangov.yml -e "ansible_ssh_private_key_file=/home/ec2-user/environment/humangov-ec2-key.pem"
application installed in all the instances

💡 Enable back the temporary credentials on Cloud9 by clicking on Settings > AWS Settings > Credentials > Turning Off the option “AWS managed temporary credentials”

Commit and push changes to AWS Code Commit

cd ~/environment/human-gov-infrastructure
git status
git add .
git status
git commit -m "Ansible configuration 1st commit plus variables.tf file changed and added nevada and florida states"

git push -u origin
pushed to git repository

Push the application source to the AWS Code Commit remote repository

~/environment/human-gov-application
git pull --rebase
git push -u origin

Capture the hands-on project implementation evidence

  • Screenshot of AWS Code Commit containing the commits
  • Screenshot of the HumanGov Application EC2 instances Up and running for 3 states on your AWS account

Fix the Ansible directory permission if needed

sudo chown -R ec2-user:ec2-user /etc/ansible

Implement VARs and Destroy the infrastructure on AWS using Terraform

export AWS_ACCESS_KEY_ID="XXXXXXXXXX"
export AWS_SECRET_ACCESS_KEY="XXXXXXXXXX/Wooc/4uPXMY"

cd ~/environment/human-gov-infrastructure/terraform
terraform destroy

For all the files used in the project find them in the below Github repository

References

https://www.digitalocean.com/community/tutorials/how-to-serve-flask-applications-with-gunicorn-and-nginx-on-ubuntu-22-04

https://www.digitalocean.com/community/tutorials/how-to-install-nginx-on-ubuntu-22-04

--

--

Sampath P

Cloud and DevOps | AWS | GCP | Microsoft Azure | Terraform | Ansible