End-to-end automated environment with Vagrant, Ansible, Docker, Jenkins, and GitLab.

Ernese Norelus
Feb 24 · 14 min read

Ansible is a beautifully simple agentless (and serverless) Configuration Management (CM) tool for configuration management, provisioning, and deployment. Its strength is the agentless nature that makes it simple and powerful. Configuration Management refers to the process of systematically handling changes to a system in a way that it maintains integrity over time, also known as idempotent. Like most of the Idempotent Infrastructure tools (no matter how many times you call the operation, the result will be the same, unless a change needs to be made.), it provides the ability through declarative language to describe system configuration; Ansible uses YAML (YAML Ain’t Markup Language) is a human-readable data serialization language syntax for expressing Ansible Playbooks. Ansible Playbooks are the building blocks for all the use cases of Ansible. The different YAML tags are classified into four types knows as declaration, vars, tasks, and handlers; through this article, we will take a practical approach in describing each one.

Getting Started

This step-by-step demo gives you a walkthrough on how to build a CI (Continuous Integration) environment with minimal effort; all the steps come together to you in the form of a use case solving a specific issue with a development environment. Apart from installing Vagrant and VirtualBox everything else is automated using Ansible Playbooks; we will go over six playbooks, each one solves a specific problem in the automation. The code is provided for you to put the demo to the test; each section builds from the previous one; it’s best that we follow the documentation systematically. Thus, I have that expectation that you will leverage the Git repository and follow along; so that a build goes well from start to end.

Many of the Ansible notions from Inventory, Playbooks, Modules, Variables, Conditionals, and Loops are covered. I have intentionally not used Roles; this is probably going to be used on a future article, as this can be somewhat complicated and overwhelming trying to pack too many concepts on the first article.

This real-life scenario will address:

  • Provisioning servers with Vagrant and VirtualBox.
  • Provisioning applications with Ansible Playbooks and Host Inventory file.
  • Continuous Integration with GitLab, Jenkins, and Docker.

Prerequisites

To follow along you are required to have these programs in place:

  • Vagrant: download the proper package for your operating system and architecture.
  • VirtualBox: by downloading, you agree to the terms and conditions of the respective license.

This article will use ubuntu/trusty64 image with version 20180404.0.0; feel free to choose your own Box by visiting Vagrantbox.es

Provisioning Servers with Vagrant and VirtualBox

We are starting with a customized configuration file for this walkthrough. The end-to-end provisioning time is in the order of 75–90 minutes depending on our your available resource, most of the time is spent provisioning GitLab, not sure why but half of it is consumed installing the server.

The configuration file can be found at ./artefacts/config/config_multi-nodes.yaml

FIGURE 1: Server Provisioning with Vagrant and VirtualBox.

Vagrant configuration file

File location: ./artefacts/config/config_multi-nodes.yaml

Building the servers with Vagrantfile

The heavy lifting happens in the Vagrantfile section, with six images to build and shared directory.

  • NFS Server
  • NFS Client
  • Docker
  • Jenkins
  • GitLab
  • Ansible

File location: ./Vagrantfile

Install Ansible repository

The ./artefacts/scripts/install_ansible.sh contains all the required instructions to:

  • Install Ansible repository
  • Install Ansible
  • Install expect, dos2unix & tree
  • Adjust timezone to be Singapore
  • Add vagrant user to sudoers
  • Generating SSH key password configuration on ansible server to later access remote servers

Provision applications with Ansible Playbooks and Host Inventory file

The playbook is the core component of any Ansible configuration. Each Ansible playbook contains one or multiple plays, which defines the work to be done for a configuration on a managed server. Ansible plays are written in YAML, as mentioned in the introduction section. In addition to Playbooks, Ansible uses an INI file as inventory source, the INI file based inventory, sections are groups or group related with special : modifiers. An administrator creates each play with environment-specific parameters for the target machines; in our case, we have a total of six that will build, the NFS server, NFS client, Docker, Java, Jenkins, and GitLab.

FIGURE 2: Application Provisioning with Ansible.

This code uses Expect; expect is a program that automates interactions with applications that expose a text terminal interface. It should be noted that using passwords as command-line arguments, like in this example, is a huge security hole; you could use Vault or other programs to encrypt the password, we aim to make the demo simple and easy to use. Our disclaimer, this is not production ready grade.

File location: ./artefacts/scripts/ssh_pass.sh

  • Adding addresses to the /etc/hosts file
  • Generating an SSH key
  • Adding the server IPs to the hosts file
  • Configuration of the INI file under /etc/ansible/hosts
  • Transferring public key

Creating six PlayBooks; each playbook is self explanatory:

  • ansible-playbook ~/artefacts/playbooks/nfs_server.yaml
  • ansible-playbook ~/artefacts/playbooks/nfs_clients.yaml
  • ansible-playbook ~/artefacts/playbooks/install_java.yaml
  • ansible-playbook ~/artefacts/playbooks/install_jenkins.yaml
  • ansible-playbook ~/artefacts/playbooks/install_docker.yaml
  • ansible-playbook ~/artefacts/playbooks/install_gitlab.yaml

File location: ./artefacts/scripts/config_ansible.sh

Instructions:

  • Installing NFS Server
  • Create mountable dir
  • Make sure the mount drive has a filesystem
  • Set mountpoints
  • Ensure NFS utilities are installed
  • Copy /etc/exports
  • Restart NFS server

File location: ./artefacts/playbooks/nfs_server.yaml

Jinja2 format File location: ./artefacts/templates/exports.j2 Ansible uses Jinja2 templating to enable dynamic expressions and access to variables as shown below:

Instructions:

  • Installing NFS Client
  • Ensure NFS common is installed
  • Create mountable dir
  • Set mountpoints

File location: ./artefacts/playbooks/nfs_clients.yaml

Instructions:

  • Install Oracle Java version 8
  • Install dependencies
  • Install add-apt-repostory
  • Add Oracle Java Repository
  • Add an apt key by id from a keyserver
  • Accept Java 8 License
  • Update cache and ignore errors in case of problems
  • Purge java 8 installer — to avoid problems if installation is repeated
  • Install Oracle Java 8

File location: ./artefacts/playbooks/install_java.yaml

Instructions:

  • Install Jenkins
  • Ensure the jenkins apt repository key is installed
  • Ensure the repository is configured
  • Install Jenkins package
  • Start & Enable Jenkins
  • Sleep for 30 seconds and continue with play
  • Get init password Jenkins
  • Print init password Jenkins

File location: ./artefacts/playbooks/install_jenkins.yaml

Instructions:

  • Install Docker
  • Ensure old versions of Docker are not installed.
  • Install list of packages
  • APT — Add Docker GPG key
  • Add Docker APT repository
  • Install Docker CE
  • Add vagrant to docker goup
  • Restart Docker

File location: ./artefacts/playbooks/install_docker.yaml

Instructions:

  • Install GitLab
  • Checking to make sure postfix is installed
  • Install GitLab repository
  • Run GitLab repository script
  • Install GitLab CE
  • Run ufw allow http script
  • Run ufw allow https script
  • Run ufw allow OpenSSH script
  • Restart with reconfigure
  • GitLab Restart with restart
  • GitLab gitlab-runsvdir status

File location: ./artefacts/playbooks/install_gitlab.yaml

Continuous Integration with GitLab, Jenkins and Docker

A well implemented DevOps (development & operations) process should mean delivering better code at a faster pace. We aimed to adopt a set of practices that automates the processes between software development and IT teams — using the available tool, such as Vagrant and VirtualBox for server provisioning and application adopting the likes of Ansible and finally embracing the development life cycle while delivering features with GitLab, Jenkins, and Docker for frequent release by fostering Continuous Integration.

  • Developers check the code locally on their computers
  • When completed — they commit changes to the repository
  • Repository sends a request (webhook) to CI system
  • CI server runs a job (tests, coverage, check syntax and others)
  • CI server releases saved artifacts for testing
  • If the build or tests fail, the CI server alerts the team
  • The team fixes the issue with constant deploy and feedback mechanism in place.
FIGURE 3: Continuous Integration with GitLab, Jenkins and Docker

Putting it all together

Issue the command below to start the automated process; depending on your server the provisioning time is in the order of 75–90 minutes for this demo to work as planned, the specifications required are 16 GB RAM and 250 GB of HDD.

Conclusion

What got my attention was the fact that Ansible uses SSH, with no agents to install on remote systems. There are many possible ways to approach this. In my attempt to learn Ansible as a Configuration Management and automation tool for Infrastructure as Code (IaC); along the way, I was able to showcase how to stand up new environments and apps in minutes. The time spent exploring the different facets of the tool and understanding its strengths and weakness in automating end-to-end environment were valuable to my learning. This article intended to show that building a sandbox is no longer a tedious task and anyone can and should try spinning one up as needed.

The complete code can be found in this git repository: https://github.com/ernesen/infra-ansible.git

References:

Follow Here for More Awesome Content

Design and Tech.Co

Ideas for the 21st Century Hustler. www.designandtech.co

Ernese Norelus

Written by

Ernese is responsible for providing technical oversight to IBM Cloud client projects

Design and Tech.Co

Ideas for the 21st Century Hustler. www.designandtech.co

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade