HumanGov: Configuration and Deployment of HumanGov SaaS Application on AWS EC2 Inventory Across US States Using Ansible Securely Storing Configuration Files on AWS Code Commit

In this project based on a real-world scenario, I used the Ansible configuration management automation to configure and deploy the innovative HumanGov SaaS application on AWS EC2 instances that will serve the entire United States.

Introduction

This project is the continuation of the HumanGov application. In the previous projects I created the code repositories and the infrastructure using Terraform for HumanGov.

In the current project I used Ansible to automate the configuration of the EC2 instances (installing all the application requirements) and the deployment of the Application.

Environment Setup

For this project was mandatory to configure Cloud 9 with non-temporary credentials because I used terraform to create the infrastructure and for some resources terraform needed admin priviledges.

The first step is to create a new user on AWS IAM with Admin privilege.

In my case I decided to call this admin user cloud9-user since I will use this user from Cloud9 environment, but the name is irrelevant.

The important part is to set AdministratorAccess to the new user.

Below you can see the user that I created.

Once the user was created, it was needed to enter in the user and generate an access key.

Before finished the creation of the access key, I donwloaded the key as a .csv file, so I can use it in later steps. Once the access key is created, it is not possible to access anymore to the key, and you will need to create a new one. So recommend you to either donwload the file or copy paste the values (the id and the secret) to some local file.

The second step is to disable the temporary credentials on Cloud9. You can achieve this by clicking on Settings > AWS Settings > Credentials > Turning Off the option “AWS managed temporary credentials”

After that you need to configure the new credentials (the access key for the cloud9-user that you created in the previous step) by using the command aws configure . The command will ask you for the key id and for the secret access key, both values appear in the .csv file that I downloaded during the creation of the key.

Update the HumanGov Terraform module

During the project on which I provisioned the infrastructure using terraform I did not had into account that the next step would be to automate the setup of the EC2 instances with Ansible.

In order to have Terraform and Ansible working hand on hand, it is necessary to do some small tweaks in the terraform configuration.

I did multiple modification on the modules/aws_humangov_infrastructure/main.tf file. I added new rules for network traffic in order to allow all inbound traffic to the ports 80 (http) and 5000 (troubleshooting port for the application server). I also allowed all traffic from the Cloud9 security group and finally I allowed all outbound traffic from the instance to the internet. In that way the packet manager can update the instance.

IMPORTANT: Do not forget to replace <YOUR_CLOUD9_SECGROUP> with the security group ID of your Cloud9 instance.

I also added the IAM role code at the end of the file.

Finally I did the following modifications in the aws_instance:

1. I Added the property iam_instance_profile to the aws_instance . In that way the instance can access to the DynamoDb table
2. I added a set of provisioners: The first provisioner adds the host to the local .ssh/known_hosts file, so when ansible connects to the EC2 instance ssh does not ask anymore if you are sure to connect. The second provisioner adds a new line in the /etc/ansible/hosts file for the aws instance. That line in the hosts file will contain the name of the instance (host) and some variables that will be used during the ansible configuration (playbook/roles). Note that the values of some variables are retrieved directly from terraform objects. The last privisioner is executed when terraform deletes the EC2 instance. The provisioner will remove the line in the /etc/ansible/hosts corresponding to the instance that it has being deleted.

resource "aws_security_group" "state_ec2_sg" {
  name        = "humangov-${var.state_name}-ec2-sg"
  description = "Allow traffic on ports 5000 and 80"

  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    from_port   = 5000
    to_port     = 5000
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    from_port       = 0
    to_port         = 0
    protocol        = "-1"
    security_groups = ["<YOUR_CLOUD9_SECGROUP>"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name = "humangov-${var.state_name}"
  }
}

resource "aws_instance" "state_ec2" {
  ami                    = "ami-007855ac798b5175e"
  instance_type          = "t2.micro"
  key_name               = "humangov-ec2-key"
  vpc_security_group_ids = [aws_security_group.state_ec2_sg.id]
  iam_instance_profile   = aws_iam_instance_profile.s3_dynamodb_full_access_instance_profile.name

  provisioner "local-exec" {
    command = "sleep 30; ssh-keyscan ${self.private_ip} >> ~/.ssh/known_hosts"
  }

  provisioner "local-exec" {
    command = "echo ${var.state_name} id=${self.id} ansible_host=${self.private_ip} ansible_user=ubuntu us_state=${var.state_name} aws_region=${var.region} aws_s3_bucket=${aws_s3_bucket.state_s3.bucket} aws_dynamodb_table=${aws_dynamodb_table.state_dynamodb.name} >> /etc/ansible/hosts"
  }

  provisioner "local-exec" {
    command = "sed -i '/${self.id}/d' /etc/ansible/hosts"
    when    = destroy
  }


  tags = {
    Name = "humangov-${var.state_name}"
  }
}

resource "aws_dynamodb_table" "state_dynamodb" {
  name         = "humangov-${var.state_name}-dynamodb"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "id"

  attribute {
    name = "id"
    type = "S"
  }

  tags = {
    Name = "humangov-${var.state_name}"
  }
}

resource "random_string" "bucket_suffix" {
  length  = 4
  special = false
  upper   = false
}


resource "aws_s3_bucket" "state_s3" {
  bucket = "humangov-${var.state_name}-s3-${random_string.bucket_suffix.result}"
  # acl    = "private"

  tags = {
    Name = "humangov-${var.state_name}"
  }
}

resource "aws_s3_bucket_ownership_controls" "state_s3" {
  bucket = aws_s3_bucket.state_s3.id
  rule {
    object_ownership = "BucketOwnerPreferred"
  }
}


resource "aws_s3_bucket_acl" "state_s3" {
  depends_on = [aws_s3_bucket_ownership_controls.state_s3]

  bucket = aws_s3_bucket.state_s3.id
  acl    = "private"
}

resource "aws_iam_role" "s3_dynamodb_full_access_role" {
  name = "humangov-${var.state_name}-s3_dynamodb_full_access_role"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
EOF

  tags = {
    Name = "humangov-${var.state_name}"
  }

}

resource "aws_iam_role_policy_attachment" "s3_full_access_role_policy_attachment" {
  role       = aws_iam_role.s3_dynamodb_full_access_role.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonS3FullAccess"

}

resource "aws_iam_role_policy_attachment" "dynamodb_full_access_role_policy_attachment" {
  role       = aws_iam_role.s3_dynamodb_full_access_role.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonDynamoDBFullAccess"

}

resource "aws_iam_instance_profile" "s3_dynamodb_full_access_instance_profile" {
  name = "humangov-${var.state_name}-s3_dynamodb_full_access_instance_profile"
  role = aws_iam_role.s3_dynamodb_full_access_role.name

  tags = {
    Name = "humangov-${var.state_name}"
  }
}

Add the variable region to the Terraform module variables file modules/aws_humangov_infrstructure/variables.tf

variable "state_name" {
  description = "The name of the US State"
}

variable "region" {
  default = "us-east-1"
}

The next step is to provision the ansible inventory in the file /etc/ansible/hosts

sudo mkdir -p /etc/ansible
sudo touch /etc/ansible/hosts
sudo chown ec2-user:ec2-user /etc/ansible/hosts
sudo chown -R ec2-user:ec2-user /etc/ansible
ls -la /etc/ansible/

In order to verify that all my changes worked as expected I decided to proceed to Provision the infrastructure with one state. In this case california :

terraform plan -var 'states=["california"]'
terraform apply -var 'states=["california"]'

Deploying the application with Ansible

In order to automate the configuration of the EC2 instances and the deployment of the application I decide to create an ansible role named humangov_webapp . You can see below the directory structure for that role.

humangov_webapp/
├── defaults
│   └── main.yml
├── handlers
│   └── main.yml
├── tasks
│   └── main.yml
├── templates
│   ├── humangov.service.j2
│   └── nginx.conf.j2
└── vars
    └── main.yml

I decided to create the role manually by using the following commnads:

cd ~/environment/human-gov-infrastructure 
mkdir ansible
cd ansible

mkdir -p roles/humangov_webapp/tasks
mkdir -p roles/humangov_webapp/handlers
mkdir -p roles/humangov_webapp/templates
mkdir -p roles/humangov_webapp/defaults
mkdir -p roles/humangov_webapp/vars
mkdir -p roles/humangov_webapp/files
ls ./roles/humangov_webapp/

After that I created the Ansible configuration file ( ansible.cfg ) inside the directory ansible . I set the following content in the file:

[defaults]
deprecation_warnings = False

After that I created the rest of the files for the ansible role by using the touch command.

touch roles/humangov_webapp/tasks/main.yml
touch roles/humangov_webapp/handlers/main.yml
touch roles/humangov_webapp/templates/nginx.conf.j2
touch roles/humangov_webapp/templates/humangov.service.j2
touch roles/humangov_webapp/defaults/main.yml
touch roles/humangov_webapp/vars/main.yml
touch deploy-humangov.yml

Then for each file I specified the content of the file.

defaults/main.yml

This file contains default values for our role variables:

---
username: ubuntu
project_name: humangov
project_path: "/home/{{ username }}/{{ project_name }}"
source_application_path: /home/ec2-user/environment/human-gov-application/src

handlers/main.yml

This file contains handlers that are triggered by tasks

---
- name: Restart Nginx
  systemd:
    name: nginx
    state: restarted
  become: yes

- name: Restart humangov
  systemd:
    name: humangov
    state: restarted
  become: yes

tasks/main.yml

This is the main tasks file where all the tasks are define. Note that some of the tasks will use the variables defined in the defaults/main.yml or in the hosts file (inventory file).

---
- name: Update and upgrade apt packages
  apt:
    upgrade: dist
    update_cache: yes
  become: yes

- name: Install required packages
  apt:
    name:
      - nginx
      - python3-pip
      - python3-dev
      - build-essential
      - libssl-dev
      - libffi-dev
      - python3-setuptools
      - python3-venv
      - unzip
    state: present
  become: yes

- name: Ensure UFW allows Nginx HTTP traffic
  ufw:
    rule: allow
    name: 'Nginx HTTP'
  become: yes

- name: Create project directory
  file:
    path: "{{ project_path }}"
    state: directory
    owner: "{{ username }}"
    group: "{{ username }}"
    mode: '0755'
  become: yes

- name: Create Python virtual environment
  command:
    cmd: python3 -m venv {{ project_path }}/humangovenv
    creates: "{{ project_path }}/humangovenv"

- name: Copy the application zip file to the destination
  copy:
    src: "{{ source_application_path }}/humangov-app.zip"
    dest: "{{ project_path }}"
    owner: "{{ username }}"
    group: "{{ username }}"
    mode: '0644'
  become: yes
  
- name: Unzip the application zip file
  unarchive:
    src: "{{ project_path }}/humangov-app.zip"
    dest: "{{ project_path }}"
    remote_src: yes
  notify: Restart humangov
  become: yes

- name: Install Python packages from requirements.txt into the virtual environment
  pip:
    requirements: "{{ project_path }}/requirements.txt"
    virtualenv: "{{ project_path }}/humangovenv"

- name: Create systemd service file for Gunicorn
  template:
    src: humangov.service.j2
    dest: /etc/systemd/system/{{ project_name }}.service
  notify: Restart humangov
  become: yes

- name: Enable and start Gunicorn service
  systemd:
    name: "{{ project_name }}"
    enabled: yes
    state: started
  become: yes

- name: Remove the default nginx configuration file
  file:
    path: /etc/nginx/sites-enabled/default
    state: absent
  become: yes

- name: Change permissions of the user's home directory
  file:
    path: "/home/{{ username }}"
    mode: '0755'
  become: yes

- name: Configure Nginx to proxy requests
  template:
    src: nginx.conf.j2
    dest: /etc/nginx/sites-available/{{ project_name }}
  become: yes

- name: Enable Nginx configuration
  file:
    src: /etc/nginx/sites-available/{{ project_name }}
    dest: /etc/nginx/sites-enabled/{{ project_name }}
    state: link
  notify: Restart Nginx
  become: yes

templates/humangov.service.j2

This is a Jinja2 template for the Gunicorn systemd service. When ansible copies the file into the web server, ansible will substitute the variables for their actual values.

[Unit]
Description=Gunicorn instance to serve {{ project_name }}
After=network.target

[Service]
User={{ username }}
Group=www-data
WorkingDirectory={{ project_path }}
Environment="US_STATE={{ us_state }}"
Environment="PATH={{ project_path }}/humangovenv/bin"
Environment="AWS_REGION={{ aws_region }}"
Environment="AWS_DYNAMODB_TABLE={{ aws_dynamodb_table }}"
Environment="AWS_BUCKET={{ aws_s3_bucket }}"
ExecStart={{ project_path }}/humangovenv/bin/gunicorn --workers 1 --bind unix:{{ project_path }}/{{ project_name }}.sock -m 007 {{ project_name }}:app

[Install]
WantedBy=multi-user.target

templates/nginx.conf.j2

This is a Jinja2 template for the Nginx configuration. When ansible copies the file into the web server, ansible will substitute the variables for their actual values.

server {
    listen 80;
    server_name humangov www.humangov;

    location / {
        include proxy_params;
        proxy_pass http://unix:{{ project_path }}/{{ project_name }}.sock;
    }
}

The last ansible configuration is to create the playbook. To use the Ansible role that I had created, I needed to create the playbook deploy_humangov.yml

- hosts: all
  roles:
    - humangov_webapp

Finally I run the playbook to verify that all my ansible configuration worked well

KEY_PAIR_FILE=/home/ec2-user/environment/human-gov-infrastructure/humangov-ec2-key.pem
ansible-playbook deploy-humangov.yml -e "ansible_ssh_private_key_file=$KEY_PAIR_FILE"

In order to test that the HumanGov app was up and running, the fastest and easiest way was by opening up the Public DNS name. I went to AWS UI and grab the Public IP from there. In my case for the california state it was:

ec2–3–81–224–202.compute-1.amazonaws.com
3.81.224.202

Provision more states in an automated way using Terraform and Ansible.

In the previous example I just provisioned the state of California, but let’s imagine that we would like to provision two more states: Nevada and Texas.

Now that the entire process is automated is as easy as going to the terraform/variables.tf file and add the states to the list of states:

variable "states" {
  description = "A list of state names"
  type        = list(string)
  default     = ["california", "nevada", "texas"]
}

After that run terraform again in order to provision automatically the required cloud infrastructure for the new states:

cd /home/ec2-user/environment/human-gov-infrastructure/terraform
terraform apply

You can verify that now terraform added new lines in the file /etc/ansible/hosts , one new line per new state that you created using terraform. Also you can see the actual values for each variable defined for each state.

cat /etc/ansible/hosts

To automatically configure the new instance and deploy the application in the new instances I needed to re-run the playbook. In this playbook run you can observe in the output how the tasks will be executed only in the newly created states.

KEY_PAIR_FILE=/home/ec2-user/environment/human-gov-infrastructure/humangov-ec2-key.pem
ansible-playbook deploy-humangov.yml -e "ansible_ssh_private_key_file=$KEY_PAIR_FILE"

I got the dns address for the newly created EC2 instances by executing terraform output

cd /home/ec2-user/environment/human-gov-infrastructure/terraform
terraform output

With the public DNS addresses, I opened new tabs in your browser and explored the app for these new states.

Summary

The present work showcases how you can automate the configuration of the EC2 instances with Ansible. It also showcases the automation of the deployment of an application by using Ansible. On top of the Ansible automation, the article also shows how to integrateTerraform with Ansible to fully automate from the provisioning of the cloud resources to the deployment of the application with just a few commands.

The article also shows some hands on practice techniques that you can use to double check that it step during the creation and configuration process is executed successfully.