Automation using Ansible

An Article by Apurva Shah

Photo by Joshua Sortino on Unsplash

One of the important aspects of web development is availability. The platform must be available to the authorized user whenever requested. However, even when all best practices are followed, there are unanticipated events that lead to downtime. During such events you have few common choices:

1. You find the cause of the downtime and till then let the platform be down. This may take from a few minutes if the problem is easy to solve, to hours if it is complicated, which incur a loss by the business.
2. You can have redundant resources. This choice would cost more as you need to have extra resources than actually required.
3. If you don’t want to spend your budget on extra resources, you would manually spawn resources and set up the environment when such a mishap event occurs. This choice would help to save the budget but may increase the risk of misconfiguration.
4. You can use Kubernetes to automate the deployment process. However, the issue with this is that it will add an overhead (setup/maintenance/expertise) and would not be suitable for an application with small/medium traffic.
5. You can have an automated script that would spawn the resources and set up the required environment whenever needed. This choice saves not only the expense of having extra resources but also helps to avoid the chances of misconfiguration lead by hasty environment setup.

In this article, we will explain how to automate the general deployment process in AWS using Ansible. Ansible is an open-source configuration management tool. Here, we will follow source-replica architecture where the source will be a Linux EC2 instance (spawned using AWS console) on which we will have Ansible installed. In this source instance, we will write a playbook that will have plays for the following:

1. Spawn a Linux EC2 instance (which will be a replica instance), create SSH to the replica, and Application Load Balancer (ALB) target group
2. Install requirements on the replica- AWS CLI, Git, Docker, Docker-compose, etc.
3. Clone the private git repository by establishing an SSH.
4. Run an application’s docker-compose.yml.
5. Create ALB and finally updating an existing Route53 record so it points to the new setup.

In order to install Ansible check out this link https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html. This playbook comprises of different plays to achieve the setup. In brief, a playbook is usually written in YAML and can consist of multiple plays, which in turn can define multiple tasks. The YAML file starts with [ — -]. In a play, we use:

• “name” — To name the play/task which is shown in the output during playbook execution.
• “hosts” — To specify names of the hosts/groups from the inventory on which the tasks are to be executed.
• “become” — A privilege escalation module or execute play as a different user.
• “var_files” — The path of the files containing values for variables used in play.
• “tasks” — They consist of the list of commands or instructions to be executed on the specified host/group.

To know more about how to work with Ansible visit https://docs.ansible.com/ansible/latest/user_guide/index.html.

PLAY — 1

In the first play of the playbook, the first task is to launch an EC2 instance using the “ec2” ansible module. Note that it is good practice to assign IAM roles to the resources, so as to have control over what it can access. Here we use the “register” ansible keyword to record the output of the task.

---
# FIRST PLAY
 - name: Launch EC2 instance, add new host, create SSH and ALB target group
   hosts: localhost
   vars_files:
    - group_vars/credential.yml    #File path where AWS credentials are stored
   tasks:
# FIRST PLAY - Task 1  
    - name: Launch EC2 Instance
      ec2:
       aws_access_key: "{{ aws_access_key_id }}"   
       aws_secret_key: "{{ aws_secret_access_key }}"
       group: ansible-sg    #Security group
       count: 1   #Number of instances to be launched
       instance_type: t2.micro   #Type of instance to be launched
       image: ami-00b243003d0e9e258    #Linux instance
       wait: true   #Wait for instance to be in required state
       region: us-east-1   
       zone: us-east-1a
       key_name: source-key    #Key name replica instance will be using
       vpc_subnet_id: subnet-12345678    #Subnet id    
       assign_public_ip: yes
       instance_profile_name: EC2Role   #IAM Role
       instance_tags:
        Name: ansible-replica   #Name for the instance
      register: ec2   #Save the returned values

The next task is to create a record of new instances in the inventory and provide necessary details that are required to establish an SSH connection. We will use the “add_host” ansible module to achieve this and use the “loop” ansible keyword to iterate through all the instances.

# FIRST PLAY - Task 2
    - name: Add new instance to host group
      add_host:
        hostname: "{{ item.public_ip }}"   
        ansible_host: "{{ item.public_ip }}"   
        groups: ansible_hosts  #Group name where instance will belong
        ansible_ssh_private_key_file: '/path/to/ansible/replica/key.pem'
        ansible_ssh_user: ec2-user   #User name that will be used while connecting
      loop: "{{ ec2.instances }}"   #Loop to add all the newly created instances

In the next task, we will use the “wait_for” ansible module. We wait for 10 seconds before checking for instance port 22 to be opened and keep checking for 320 seconds until its available.

# FIRST PLAY - Task 3   
    - name: Wait for SSH
      wait_for:
       host: "{{ item.public_dns_name }}"
       port: 22
       delay: 10   #Wait for 10 seconds before checking port availability
       timeout: 320  #Keep checking  for 320 seconds
       state: started   #Until port is open
      loop: "{{ ec2.instances }}"

The last task is to create a target group for our ALB using the “elb_target_group” ansible module, so it can route traffic to our newly created instance.

# FIRST PLAY - Task 4
    - name: Create ALB target group
      elb_target_group:
       aws_access_key: "{{ aws_access_key_id }}"
       aws_secret_key: "{{ aws_secret_access_key }}"
       name: "ansible-target-group"   #Name for the target group
       target_type: instance
       protocol: "http"
       region: "us-east-1"
       port: 80
       vpc_id: vpc-12345678    #VPC ID
       targets:
        - Id: "{{ item.id }}"
          Port: 80
       state: "present"
      loop: "{{ ec2.instances }}"   #Include all newly created instances

Finally, if we put all the tasks together we get our first play. A common error that can be faced while running this play is “msg”: “boto required for this module”

One of the solutions is to add the following in the “hosts” file, this solution was found at this forum https://github.com/ansible/ansible/issues/15019

[localhost]
localhost ansible_connection=local ansible_python_interpreter=python

PLAY — 2

In the second play, we install the required configuration on the new instance. Note that now the value of “hosts” is “ansible_hosts” (group name specified while recording new instance in inventory). For installing the requirements, we can either use ansible module like yum, apt, etc. or use terminal command and pass them as value to the “command” ansible module. One of the differences is that the “yum/apt” module is more efficient for installing packages than using the “command” module to do so.

# SECOND PLAY 
 - name: Install requirements on EC2 instance
   hosts: ansible_hosts
   gather_facts: True
   become: True   #To allow privilege escalation  
   tasks: 
    - name: Install awscli
      command: pip install --upgrade awscli
    - name: Install docker   #Install docker using the usual command 
      command: yum install docker -y
    - name: Install git   #Install git using yum ansible module
      yum:
       name: git   #Install git using yum
       state: latest   #Latest version

We can also use the “shell” ansible module to run multiple commands through the shell as follows

- name: Install docker-compose
      shell: |
       curl -L "https://github.com/docker/compose/releases/download/1.20.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
       chmod +x /usr/local/bin/docker-compose
       group add docker
       usermod -aG docker ec2-user
       export DOCKER_HOST=127.0.0.1:2375
       service docker start

Now, by combining both the snippets we get a complete second play. An error that was faced while running this play was

ERROR: Couldn't connect to Docker daemon at http+docker://localunixsocket - is it running?
If it's at a non-standard location, specify the URL with the DOCKER_HOST environment variable.

The error was solved by adding the following, found at the forum https://github.com/docker/compose/issues/4181

usermod -aG docker ec2-user    #Add docker group to current user
export DOCKER_HOST=127.0.0.1:2375   #Explicitly set docker host

PLAY — 3

The third play will clone the private git repository as follows,

# THIRD PLAY 
 - name: Clone git repository
   hosts: ansible_hosts
   tasks:
    - name: Copy git SSH keys
      copy: src=/home/ec2-user/.ssh/ dest=/home/ec2-user/.ssh/ mode=0700
      sudo: true
    - name: Change directory permission
      command: sudo chmod -R 700 /home/ec2-user/.ssh/ 
    - name: Enable SSH agent
      shell: |
       eval "$(ssh-agent)"   #Start SSH agent
       export SSH_AUTH_SOCK   
       export SSH_AGENT_PID
       ssh-add
    - name: Clone a private repository
      shell: git clone git@gitlab.com:path/to/application.git
    - name: shred git keys
      shell: |
       shred -zuv -n 5 /home/ec2-user/.ssh

The pre-requisite for this play is to have git SSH keys in the “~/.ssh” folder. First, we copy SSH keys to replica using the “copy” ansible module and then set up permission. Note that if you are copying individual files (from source-ansible) then the destination (on replica) must also be a file. Here we are coping folder (from source-ansible) so even the destination (on replica) is a folder. Then, we enable and create an SSH connection to clone the git repository. Finally, we permanently remove SSH keys from the volume store using “shred”. Note that just by using the “rm” command would not be sufficient as the keys are still present on the storage volume until that space is overwritten by some other data. So by using “shred” we delete and overwrite the space utilized to store keys with zeros.

PLAY — 4

Next play will traverse to the directory containing the docker-compose.yml file and do docker-compose up.

# FOURTH PLAY
 - name: Run the application
   hosts: ansible_hosts
   gather_facts: True
   tasks:
    - name: Do docker-compose up
      shell: |
       cd /home/ec2-user/git-repository/   #Path where docker-compose.yml is present
       docker-compose up -d

PLAY — 5

The final play first creates an ALB with HTTP and HTTPS listeners using the “elb_application_lb“ ansible module as follows

# FIFTH PLAY
 - name: Create ALB and modify the existing Route53
   hosts: localhost
   vars_files:
    - group_vars/credential.yml
   tasks:
# FIFTH PLAY - Task 1  
    - name: Create ALB
      elb_application_lb:
       aws_access_key: "{{ aws_access_key_id }}"
       aws_secret_key: "{{ aws_secret_access_key }}"
       name: ansible-alb
       region: us-east-1
       security_groups:
        - ansible-sg  
       subnets:
        - subnet-12345678     
        - subnet-91011121
       listeners:
        - Protocol: HTTP 
          Port: 80 
          DefaultActions:
           - Type: forward 
             TargetGroupName: ansible-target-group   #Same name given while creating it
        - Protocol: HTTPS
          Port: 443
          SslPolicy: ELBSecurityPolicy-2016-08
          Certificates:
           - CertificateArn: arn:aws:acm:us-east-1:123:certificate/456  
          DefaultActions:
           - Type: forward 
             TargetGroupName: ansible-target-group 
       purge_listeners: no
       state: present
      register: elb

The next task then updates an existing Route53 record. Note that the value of the “value” parameter in the “route53” ansible module is not just the DNS name of ALB, rather it is a concatenation of the word “dualstack” and the DNS name of ALB.

# FIFTH PLAY - Task 2
    - name: Modify A record in route53
      route53:
       aws_access_key: "{{ aws_access_key_id }}"
       aws_secret_key: "{{ aws_secret_access_key }}"
       state: present   #Existing record
       overwrite: yes   #Overwrites the existing record if found
       zone: grad4-demo.com
       record: grad4-demo.com
       type: A
       value: "dualstack.{{ elb.dns_name }}"  #concatenates dualstack and ALB DNS name
       alias: True
       alias_hosted_zone_id: "{{ elb.canonical_hosted_zone_id }}"

Now combine both tasks to get the last play. Finally, to execute the playbook we can use the command

ansible-playbook <playbook-name.yml> -vvvv

Here, the number of v’s depend on how detailed the output you need. You can also dry run (show the expected output without actually running it) the playbook to check for errors using either of the following commands

ansible-playbook <playbook-name.yml> --syntax-check 
ansible-playbook <playbook-name.yml> --check  
ansible-playbook <playbook-name.yml> -C

Notes:

• After installation, if you don’t get any ansible files/folders, create a folder named “ansible” at the “/etc” location. After creating this folder, create an “ansible.cfg” file (where configuration setting can be defined), a “hosts” file (it is inventory). If you want to setup ansible vault you can visit the link https://serversforhackers.com/c/how-ansible-vault-works (If you don’t want to pass the vault password file path while executing playbook, you can set “vault_password_file = /path/to/vault_password_file“ in ansible.cfg)
• In order to completely erase the git keys from the store volume, we have used the “shred” library. However, “shred” might not be the best option for other configuration systems as it depends on the way the file system overwrites data.

GRAD4

At GRAD4, we use the above-described technology to avoid any unanticipated platform downtime so that it is always accessible to the authorized user. We also use it to manage and patch our systems on a regular basis, as well as to have a secure environment configuration during any unfortunate event by avoiding any system misconfiguration. We aim to provide the best customer service. To achieve it, we follow best-recommended practices and also incorporate new technologies to avoid creating any inconvenience to our users and also enhance user satisfaction.