Deploy a MongoDB Replica Set in a DevOps fashion style (Infrastructure as Code with Terraform)

40 min readApr 12, 2019

INTRODUCTION

Hello again, its been a Journey since I wrote my last article back in 2017, it has been a very busy years, I have been moving all around, I moved from one city to another, and from that city to another, from one company to another, but along the journey I have learned a lot of new technologies, new job roles, processes, gained friends from different places cultures and countries, it has been quite interesting, So for the ones who doesn’t know me, my name is Cristian Ramirez (CR), i’m a DevOps Engineer, Cloud Expert and Sr. Software Engineer, I’m currently working on SAP Ariba with one of the most cutting edge DevOps Strategies using all Hashicorp Stack, previously I worked as Software Developer using Golang building Microservices and Plugins for Terraform, also I am part of the fastest growing Silicon Valley StartUp, DigitalOnUs, we are Top 3 premium partners with Hashicorp and we are also partners with Docker, SAP and other good tech companies, thats a little brief intro about myself. So now the fun part, i am going to upgrade my article How to deploy a MongoDB Replica Set using Docker in this new article applying DevOps best practices and using cutting edge technologies.

@cramirez92

SPECS

This article is going to be a walk-through in how to set up a MongoDB replica set with authentication, we are going to use the following toolset:

Packer v1.3.1
Ansible latest
Terraform v0.11.10
Docker 18.06.0-ce
MongoDB 4.0.7
Bash Scripting

We are going to use aws as our public cloud, in order to follow up this article you will need to have an aws account and the previous tools mentioned installed locally except for mongodb and docker.

Disclaimer:

Some of the resources deployed in AWS may cost money.

CLOUD INFRASTRUCTURE ARCHITECTURE

ARCHITECTURE OF REPLICA SET

So the result of this article will be the above architectures of our MongoDB Cluster using our toolset already mentioned, and it will be created with tons of automations.

PRE-REQUISITES

This is a DevOps exercise, i made this article the most friendly as possible so you can easily followup.

Note: If you want to followup this exercise as it is, you may need to have an aws account, and the Hashicorp binaries installed locally. This may generate a cost for your aws account, but this can also be done on a free tier account.If you’re not using an account that qualifies under the AWS free-tier, you may be charged to run these examples. The charge should only be a few cents, but I will not be responsible if it ends up being more. If you are ready to go and have an aws account you will need also to create an aws key pair so you can use this pem file to ssh to the instances that we are going to create in aws.

With no more to say, lets go get our hands dirty with some DevOps and very cool technologies.

Our pipeline is divided in 3 steps with 3 subtask each step.

STEP 1 will do the following:

Create a host image
Create aws network
Create aws EC2 instances

STEP 2 will do the following:

Configure mongodb files
Create mongodb volume and container for pre-configuration
Create final mongodb container and init replica set and start configuration automatically

STEP 3 will do the following:

Executing terraform commands
MongoDB configuration verification
MongoDB queries just for fun

#STEP 1

A) CREATE HOST IMAGE FOR MONGODB SERVER

In order to spin up our mongodb cluster, first we need to configure our instances and will be doing it even without running or creating any instance, how is this going to work? Well here is where Packer comes into the picture, with Packer we are going to configure our instance from which base OS to all the necessary tools that our mongo servers will be running.

But first why do we need to do this ?

Because we cannot have mutable Infrastructure, speaking in a figurative way lets say that we have mutable infrastructure having a fleet of many hundreds or many thousands of machines running mongodb or any other software like a web server (Nginx, apache, etc) for different projects, microservices, etc. and they all fail in slightly different ways, and then you will end up fixing and upgrading your servers in place and have them on different shapes with different software versions running so you may could end up having 10 servers running version 1 and other 5 running version 1 + patch 1, and other 5 running version 1.2 + patch 2, and these then become incredibly hard to debug because your system is in a poorly understood state.

An immutable infrastructure approach

The difference is, when we go immutable, we don’t want to ever upgrade our servers in place. Once the server exists, we never try to upgrade it to V2. What we will do is to create our server with Packer, call it version 1, we’ll install the necessary software and we’ll take a snapshot of this image. Then we boot our servers with this version, we allow user traffic to start coming into our server, and if we detect some issues or failures on our version 1, what we are going to do is to create brand new servers with version 2 image, so this are distinct machines, we are not trying to upgrade the existing and running infrastructure.

So for V2 image server If there’s any error, we will abort this server creation, but if we successfully create version 2, and there were no errors and everything is installed, then what we will do is to switch traffic over that allows, users to starts hitting version 2 instead of version 1. Then we will just decommission version 1 and we take the version 1 fleet out of production, and we have successfully achieved our goal, that is to never try to in place modify the infrastructure. And with this, we have a discrete notion of versioning, there was either version 1 running and that’s where traffic went, or there was version 2 running and that’s where traffic went. There was no version 1.5 or 1.2 or 1.x in between.

The advantage of this becomes, as we think about risk and complexity, there’s much lower risk, because we don’t have these undefined states that aren’t validated, but we also reduce the complexity of our infrastructure.

So the correct approach to do this is following the immutable approach, because there are constant changes to server setups, new software to be installed, packages upgraded, old software versions removed, so now lets see how we do all what I have talked above.

The stack for building our host image is going to be Packer and a series of shell scripts for provisioning. This works well for a small team or one person in charge of this, but as teams always grows and more people are involved and they are making changes to the scripts, this can easily get out of hand and become confusing, so the perfect complement for building our image is Ansible.

Packer provides a single workflow to package applications for any target environment.

Ansible with it’s YAML based syntax and agentless model fits quite nicely. According to their website “Ansible is the simplest way to automate IT”. You could compare it to other configuration management systems like Puppet or Chef. These are complicated to setup and require installation of an agent on every node. Ansible is different. You simply install it on your machine and every command you issue is run via SSH on your servers. There is nothing you need to install on your servers and there are no running agents either.

Something that i appreciate the most is the fact that Ansible playbooks (the pendant to Chef cookbooks or Puppet modules) are plain YAML files. This keeps the playbooks simple and easy to understand. (Try writing complicated shell commands with multiple levels of quoting and you will see what I mean.) Ansible is easy to understand even for somebody who doesn’t know a lot about Ansible. For a more thorough introduction, please see the Ansible homepage and don’t forget to check docs available at http://docs.ansible.com.

So lets see how our Packer file look like:

"variables": {"aws_access_key": "{{env `AWS_ACCESS_KEY`}}","aws_secret_key": "{{env `AWS_SECRET_KEY`}}","aws_region": "us-west-1","instance_type": "t2.large","ami_name_prefix": "ubuntu16.04-t2","provision_script": "provision.sh","playbook": "ansible/{{env `MACHINE_TYPE`}}.yml","machine_type": "{{env `MACHINE_TYPE`}}"}

The first section is variables here we are declaring the variables that we are going to use along the building process (in the packer file only, not vm variables). Here we are reading the env variable from our local computer where we are running packer for aws authentication and machine type, since we are including ansible and we can have different roles we can have different ansible play-books lets say, db playbook, web playbook, network playbook, etc. so since the begging of this process we are automating the creation of our host image, just by selecting a playbook defined in ansible, and in our case machine type will be mongodb, we will see later on why we are doing it like this.

Next section is builders:

"builders": [{"type": "amazon-ebs","access_key": "{{user `aws_access_key`}}","secret_key": "{{user `aws_secret_key`}}","region": "{{user `aws_region`}}","source_ami_filter": { "filters": { "virtualization-type": "hvm", "name": "ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-server-20190212",  "root-device-type": "ebs"}, "owners": ["099720109477"], "most_recent": true},"instance_type": "{{user `instance_type`}}","ssh_username": "ubuntu","ami_name": "{{user `machine_type`}}-packer-{{timestamp}}","ami_description": "Ubuntu 16.06 {{user `machine_type`}} Host Image","launch_block_device_mappings": [{ "device_name": "/dev/sda1", "volume_size": 50, "volume_type": "gp2", "delete_on_termination": true}],"tags": { "Name": "aero-{{user `machine_type`}}-host", "role": "aero-{{user `machine_type`}}-host"},"run_tags": { "Name": "aero-{{user `machine_type`}}-host", "role": "aero-{{user `machine_type`}}-host"}}],

In the builders section, since we are using aws, here we set the characteristics the base image that will create our snapshot, we select the base OS, volume disk configs, aws tags, basically here we set all aws ec2 configurations.

And finally the provisioners section:

"provisioners": [{"type": "file","source": "ansible","destination": "/var/tmp/"}, {"type": "file","source": "roles","destination": "/var/tmp/ansible/roles"}, {"type": "file","source": "provision.sh","destination": "/tmp/{{user `provision_script`}}"}, { "type": "shell", "inline": [  "chmod u+x /tmp/{{user `provision_script`}}",  "/tmp/{{user `provision_script`}} {{user `playbook`}}" ]}]}

In the provisioners section we have 2 types of provisioning: “file” and “shell”, for file what we are doing is copying over our ansible script to the ec2 machine that will be our base image, then for the shell, we are going to execute shell script and this is basically installing ansible and finally execute the ansible playbook to install our mongo configurations. So lets take a look at our provision.sh script before I start to talk about our ansible scripts. This script will basically install ansible in our ec2 instance and then execute our ansible playbook.

Lets talk now about Ansible, our folder structure is divided in ansible: that contains the playbooks, roles: that contains the different packages and configurations that we are going to do to our mongodb server, and inside each role we have tasks, files and templates. And this is what it looks like:

.├── ansible│   ├── group_vars│   │   └── all│   └── mongodb.yml├── aws.json├── provision.sh├── roles│   ├── cloud│   │   ├── files│   │   │   ├── aws│   │   │   │   └── metadata-provider.sh│   │   │   ├── cloud-service.sh│   │   │   ├── cloud.service│   │   │   └── gce│   │   │       └── metadata-provider.sh│   │   └── tasks│   │       └── main.yml│   ├── consul-template│   │   └── tasks│   │       └── main.yml│   ├── docker│   │   ├── files│   │   │   └── docker-tables.sh│   │   └── tasks│   │       └── main.yml│   ├── mongo│   │   ├── files│   │   │   ├── bookstore/│   │   │   ├── cinemas/│   │   │   ├── mongo-keyfile│   │   │   ├── mongo-service.sh│   │   │   ├── mongo.service│   │   │   ├── pre-start.sh│   │   │   ├── primary-db-only.sh│   │   │   └── start.sh│   │   ├── tasks│   │   │   └── main.yml│   │   └── templates│   │       ├── admin.js.j2│   │       ├── grantRole.js.j2│   │       └── replica.js.j2│   ├── os│   │   ├── files│   │   │   ├── common-iptables.sh│   │   │   └── iptables-helper.sh│   │   └── tasks│   │       └── main.yml│   └── python│       └── tasks│           └── main.yml└── start.sh24 directories, 36 files

I will assume that you have basic understanding about Ansible, the following roles are the ones that we are going to install in our image, the output of our mongodb.yml file is the following:

---# Installation steps- hosts: allroles:- python- os- cloud- consul-template- docker- mongo

So why do we need python for, we need to install the aws-cli in order to get the ip’s of all our db instances, the cloud role has some bash functions to detect what type of cloud we are using, consul-template to dynamically generate our mongodb files, docker-role to install docker in our image and finally mongo role has all the configurations that are necessary for our mongodb replica set. I am going to explain in more detail the mongo role later on in this article.

In order to execute Step 1A, we need to have packer installed and execute the following command:

$ bash start.sh

and the content of our `start.sh` script is the following:

Which basically is calling the packer binary, inside of our buildImage function, and there are two things happening here, buildImage function is receiving two arguments the first one is to set the machine type and this machine type if you remember, is going to select the correct playbook inside our ansible folder, and the second argument is the cloud type we are going to use and in this case we have only create one packer configuration for aws; if we had a gcp packer configuration we just need to change the second argument and that’s it or we can create both, this is where automation is so cool.

So to conclude Step 1A, we are going to create our image in aws using Packer + Ansible with all the configurations necessary that we need to create a mongodb replica set cluster in an automated fashion way.

NOTE: before creating an aws or gcp or any cloud or bare metal image, I strongly recommend to test all the configurations in a local environment and here is where Vagrant comes into the picture of using another Hashicorp tool, once we test our configurations in Vagrant we are good to go and build our image to roll it out into the cloud or into production.

B) CREATE AWS INFRASTRUCTURE WITH TERRAFORM

Terraform is a product to provision infrastructure and application resources across any cloud infrastructure using a common workflow. Terraform enables operators to safely and predictably create, change, and improve production infrastructure. It codifies APIs into declarative configuration files that can be shared amongst team members, treated as code, edited, reviewed, and versioned. This APIs are a series of provider to support cloud types such as AWS, GCP, Azure, vSphere, Nutanix(I contribute for this provider development) , Outscale(I contribute for this provider development), Alibaba Cloud, etc.

Provisioning requires infrastructure-specific images because there is no common packaging format for virtual machines across providers. HashiCorp Packer enables operators to build many machine image types from a single source, and we have made this in section (A) as well as we automate our image creation to support this many image type of clouds. Then a Terraform configuration can reference these images to provision infrastructure using the cloud-specific images created by Packer.

So now that we have an understanding about what terraform is let’s see how we are going to implement it in our architecture.

Before we begin to create our mongodb instances we need to defined a scope in our aws cloud, we need to create a protected zone that will have private access and a DMZ zone that will have public access, and in order to have that first we need to create our network infrastructure, and we need the following as established in our architecture diagram:

1 aws vpc
1 public subnet and its route tables
1 private subnet and its route tables
1 internet gateway
1 nat gateway
1 network load balancer ( Optional )

Why do we need this components ?

A virtual private cloud (VPC) is a virtual network dedicated to your AWS account. It is logically isolated from other virtual networks in the AWS Cloud. You can launch your AWS resources, such as Amazon EC2 instances, into your VPC. You can specify an IP address range for the VPC, add subnets, associate security groups, and configure route tables.

- Amazon

Basically we need an aws vpc in order to create our own workspace (if want to look the vpc like that), so we can manage all our resources that we are going to create inside that workspace, and once created we need to create 1 Public Subnet and this subnet will be our DMZ so we can deploy our bastion host here, 1 Private Subnet this is our protected zone to host our database instances, and since our databases are going to be in the protected zone this instances will not have public ip’s, this instances will not have neither access from the internet nor access to the internet, and here is where the Nat Gateway comes into the picture, in order that our instances can have access to the internet to whatever we need them to access the internet (pull docker images as an example), but we won’t have access from the internet to our instances, then in our Public Subnet our bastion host will have a public ip, so we can ssh into this instance and from here we can jump Into our database instances, and since this is in our public subnet and if we want that this instance have access to internet for whatever reasons, then we need an internet gateway, this is the description of our network infrastructure so now lets see how does this looks in code.

This is our terraform folder structure:

.├── bin│   └── user-data-db.sh├── main.tf├── modules│   ├── instance│   │   └── main.tf│   └── network│       └── aero-bastion│           ├── main.tf│           ├── output.tf│           ├── sg.tf│           ├── subnet.tf│           ├── variables.tf│           └── vpc.tf├── network.tf├── query.tf├── terraform.tfstate└── variables.tf5 directories, 13 files

Here we can see that we have 1 module that is for creating a bastion network and 1 for creating instances, modules in terraform are used for reuse terraform code in different places so we don’t have to write the same code always, modules is DRY (don’t repeat yourself) as we know it programming languages, the variables.tf file is where we are going to declare all the variables that we are going to use across all the terraform configurations, this files are limited to its folder scope, we can set default values or it can be dynamic values, and I will show you both cases, as you can see in the bastion network folder we have another variables.tf that belongs to that scope and that file will have variables with dynamic values set, so first lets take a look to this files in the order that I mentioned in the begging of the section B.

Note: In order to use terraform we need to have a basic knowledge about how the cloud works, and in this case how aws works, what kind of resources does aws has, what are the attributes of the resources, what kind of values do we need set, If you are just begging to know how the cloud providers works I’ll suggest you to take first some basic courses about aws or gcp, or your cloud of preference so you can have an overall understanding about how the cloud works, then go to the Terraform documentation select your provider if available and take a look and see what are the resources available for terraform, and finally you can start playing with terraform.

So first we need to create our vpc and it requires an ip range in order to create our network, in this case the range is set in the cidr_block, and as you can see we are using the variables that are defined in the variables.tf, we need the vpc to have dns, to be able to discover our instances via dns. Next we set some tags to our resource, this is a best practice to follow because with tagging our resources with the correct information we can know where does each resource belongs to, to which scope it belongs to like dev | qa | e2e, which terraform workspace has been created, etc. Next we need to create our internet gateway so that our resources inside our vpc can have access to the internet, and we need to attach our internet gateway to our newly created vpc, then we need to create an elastic ip, aws will assign a public ip to this resource and this public ip we will attach it to our nat gateway, that is the final resource that we are going to create, the nat gateway will provide access to internet to the resources inside the private subnets.

vpc.tf

Continuing with our network infrastructure development now lets create the subnets, as I mentioned earlier we need 1 public and 1 private subnet, there is no such thing or attribute that makes one public or private, the difference between this subnets is the way how we are going to treat this subnets. First we our going to create our private subnet, for this we need to specify again an ip range and in this case it should be an ip range that belongs to the vpc_cidr_block that we defined earlier, next we need to set an availability zone; but why do we need to specify an availability zone?

Amazon cloud computing resources are hosted in multiple locations world-wide. These locations are composed of AWS Regions and Availability Zones. Each AWS Region is a separate geographic area. Each AWS Region has multiple, isolated locations known as Availability Zones. Amazon RDS provides you the ability to place resources, such as instances, and data in multiple locations. Resources aren’t replicated across AWS Regions unless you do so specifically.
Amazon operates state-of-the-art, highly-available data centers. Although rare, failures can occur that affect the availability of instances that are in the same location. If you host all your instances in a single location that is affected by such a failure, none of your instances would be available.

and finally we need to attach the private subnet to the vpc created earlier. Next to create the public subnet we need the same configuration as the private subnet, but the only change we need to do is to set a different ip range, so as I mentioned before there is no such attribute that makes one subnet public or private, we make them by the way we are going to implement them and use this subnets. Once we have defined our subnets now lets create our route tables, but wait, what are this route tables and why de we need them for ?

A route table contains a set of rules, called routes, that are used to determine where network traffic is directed. Each subnet in your VPC must be associated with a route table; the table controls the routing for the subnet. Your VPC can have route tables other than the default table. One way to protect your VPC is to leave the main route table in its original default state (with only the local route), and explicitly associate each new subnet you create with one of the custom route tables you’ve created. This ensures that you explicitly control how each subnet routes outbound traffic.

Once we have created our route tables for each subnet, we need to associate this route tables, and here is how we mark our subnets as private or public, because we are giving them a purpose and restrictions, so for the public subnet route table we are going to associate it with the internet gateway and for the private subnet we are going to associate it with our nat gateway, and this is how the terraform subnet definition looks like.

subnet.tf

Next is the security groups.

A security group acts as a virtual firewall for your instance to control inbound and outbound traffic. When you launch an instance in a VPC, you can assign up to five security groups to the instance. Security groups act at the instance level, not the subnet level. Therefore, each instance in a subnet in your VPC could be assigned to a different set of security groups.

First we are going to create a SG that will be attached to the public subnet where we are allowing only traffic to the ssh protocol which runs on port 22, and also we are allowing icmp protocol for testing purposes only, then we specify our vpc to attach this SG and what would be the ip range that we are allowing to communications to this SG, but this time we will put the 0.0.0.0/0, since this would be a public subnet and we are allowing traffic from the internet. Then we are going to create the private SG where we are allowing traffic only to the ssh protocol and to the mongodb protocol which runs on ports 22 and 27017, and we are allowing outbound communications to internet from our private SG , and anything else would be closed. So lets look how is this defined in terraform:

We have created our network infrastructure so we are done until here, but I would like to create another subnet, I want this subnet to be specifically for our mongodb replica set, and as you read earlier, we can associate any number of subnets to a route table, so we are going to associate our subnet to our private route table created earlier, so that our instances can have access to the internet through the nat gateway, and our terraform code looks like the following:

db-network.tf

C) CREATE AWS EC2 INSTANCES

So now we have arrived to the section where we are going to create our instances using the image created by Packer, configured by Ansible and finally we are going to provision it with Terraform in our AWS Cloud in our newly fresh network infrastructure created … uff lots of stuff has happened until now and the best is yet to come.

So to complete our architecture defined at the begging, we need to create 1 bastion host instance in the public subnet and 3 ec2 instances that will run a mongodb containers in the db-private subnet. Earlier i mentioned a little bit about what Terraform modules are for, but what really is the benefit of using this modules ?

A module is a container for multiple resources that are used together. Modules can be used to create lightweight abstractions, so that you can describe your infrastructure in terms of its architecture, rather than directly in terms of physical objects.
Re-usable modules are defined using all of the same configuration language concepts we use in root modules. Most commonly, modules use:
Input variables to accept values from the calling module.
Output values to return results to the calling module, which it can then use to populate arguments elsewhere.
Resources to define one or more infrastructure objects that the module will manage.
To define a module, create a new directory for it and place one or more .tf files inside just as you would do for a root module. Terraform can load modules either from local relative paths or from remote repositories; if a module will be re-used by lots of configurations you may wish to place it in its own version control repository.

So since we are going to create 4 aws EC2 instances, How do we avoid code duplication for this ?

The solution is to package the code as a Terraform module.

Lets see how the terraform code looks like to create an instance. Our terraform code a the begging is declaring the variables that we need to provide to be able to use it inside our module, we can split this file into two, one for variables and the other one for resources, but since is a very short definition we can keep it in a single file. And to create our instance we just need to provide the required properties to create an aws EC2 instance with the values that we are providing dynamically, this values can have default values also if we don’t want to provide dynamic values, so next step is how we actually implement this module.

And as simple as that, we just call our module to create our 4 instances: 1 Bastion Host and 3 Mongo Servers, so now we have concluded STEP 1 with its 3 subtasks, but there is something missing to explain here, there are to many values that are coming from else where to fill the module variables values; we can see that we are filling the values in 4 different ways:

1) hard coding the values

2) calling ${ data.x.x }

3) calling ${ module.x.x }

4) calling ${ var.x }

From where are this values coming from ? Well all ${var.x} is coming from the variables.tf file that we defined in at the root of project which is also known as the root module, this variables are most commonly defined with known values or it can come from the command line once we run the terraform binary, it will prompt you for the value. I will show this later on in this article, the ${ data.x.x } this values are coming from the terraform datasource object, we basically query the api of our provider in our case aws, and we ask for some values there, as an example we are querying for the mongo instance image that we create with packer, we just need to give the correct query values so that the datasources can bring us the image id, and we can use it inside our instance module, and the ${ module.x.x } this values are coming from other modules in our case we create a network module for our aws network infrastructure, and there we defined some output values that can be used to implement in other resources outside the module, in our case we need the subnets ids and security groups. So this is how our main file looks like:

Next I want to talk about the query.tf file which we are using to create datasource objects, here we are retrieving our mongo server image, the availability zone and we are doing something else with one datasource object: the “template_file” this datasource let us read the a bash script file and also let us inject variables into that file, in our case we are injecting some database credentials and also our aws keys, but why we are doing this ? This is the worst approach of doing this, but I wanted to show you that we can inject values inside this files, and this will be read by our instances once they are up and running; So I am going to explain in STEP 2 how are we going to use this values and why do we need them; Also in this article I don’t want to showcase the security part and secrets management, I will dedicate a second article about how to do secrets management and what are the best practices and for this I will introduce another great tool from Hashicorp and this is Vault, so please stay tuned for this next article.

query.tf

Now let me show you how actually we can read the variables injected from the “template_file” datasource, inside the bash script file, and here we are exporting this variables into an environment file, once again this is not the recommended way it is only for demonstration purposes, we will update this once Vault comes in into our architecture. And at the end of this script we are just basically starting our mongo service, this mongo service I will explain it in STEP 2.

user-data-db.sh

Now we are ready to create the infrastructure designed in our architecture diagram, just by running “terraform apply” but I haven’t explained everything yet, so I suggest not to create the infrastructure yet, until we cover the second step, so we can understand what is actually happening in the background, until now we have covered all the automation for the mongodb replica set infrastructure, but there is more automation to see and that is the mongo configuration, so lets take a look at.

#STEP 2

A) AUTOMATED MONGO CONFIGURATION

So now lets get back to ansible configurations, now lets dive in into the mongo role that will configure the mongo server instances. The folder structure of the mongo role is the following:

bookstore and cinemas folders are mock data to have it handy to test mongodb queries.
the rest of the files are bash configurations to create our mongodb containers and replica set configurations.

.├── files│   ├── bookstore/│   ├── cinemas/│   ├── mongo-keyfile│   ├── mongo-service.sh│   ├── mongo.service│   ├── pre-start.sh│   ├── primary-db-only.sh│   └── start.sh├── tasks│   └── main.yml└── templates├── admin.js.j2├── grantRole.js.j2└── replica.js.j25 directories, 19 files

NOTE: this is almost a production-ready configuration, as I said almost because I am hardcoding passwords and security configurations, I did this for simplicity and to have a better understanding of this pipeline, the configurations and the processes itself. I will create a second article focused on using Vault for all the security related configurations, so stay curious and pending once i released the security part for this mongodb replica set.

Ansible here is only playing one role, it only creates folders in the base instance and copies the bash files that has the mongo configurations, the main.yml file is pretty straightforward to understand.

The execution order of the bash files are the following:

mongo.service: this is a system service to init the mongo configurations and this is called from the user-data.sh file that we use in the terraform code.
mongodb-service.sh: this is a script to configure all the files and env variables dynamically and is called from the mongo-service systemd script.
pre-start.sh: this is a script that makes pre configurations for the mongo containers and is called from mongodb-service.sh to do pre configurations setup.
start.sh: this is a script that start the final mongo containers with all the configurations done and is called from mongodb-service.sh once pre-start.sh is done.
primary-db-only.sh: is called from mongodb-service.sh once start.sh is done, and it will configure the replica set.

So now I will explain the files mentioned above in that order to see what is happening.

mongo.service is a systemd script it doesn’t has nothing to fancy is just calling the mongodb-service.sh so is straightforward to understand.

Now lets take a look at the mongodb-service.sh in 3 parts.

At the begging of this files we are going to create all the environment variables that we need to use across all the configuration files, first we need to get the IP address of the instance as an env var to have it handy, then the mongodb port and couple of database credentials, this will be used to create the database users for the test databases (cinemas and bookstore). Then DB_HOST var which will be the name of the container, and as you can see we are setting all this variables in our /etc/environment file so we can call this file when we need to make use of our variables only when we need them, so we don’t populate the instance with to much env variables.

Next we are going to query aws using the awscli to get the list of ip’s of the instances created that will be part of the replica set, and here is where we are making use of the access_key and secret_key values, if you remember we inject this variables using terraform, so that was the reason of doing that, and finally export this ip’s in our environment file, to use this vars when the configuration needs them.

Then we are going to create mongo files dynamically with consul-template, the following files are used to create the admin user for the database, grant permission to the admin user, and create the replica user for the replica set configurations. For more information about the mongodb replica set please visit the mongodb documentation.

And at the end we are creating a file called host.sh that has a function inside, which basically returns a string with — add-host docker flag with all the ip’s that we query from the awscli, so that our containers can have in their resolve.conf the host and ip of the 3 instances that we are creating, we are going to see later on where and how is this implemented.

Quickly lets view how a template from ansible config looks like; actually is not an ansible template we just put it as a template for demo purposes, instead this are consul-template files. This templates are basically reading the environment variables that we have declared previously steps using the consul-template syntax. Later on I will explain how to do the same thing but instead of reading the env vars hardcoded this config will read from vault. Actually the values of this env variables are coming from terraform variables if you can remember the content of the variables.tf in the root module.

And finally at the end of the mongodb-service.sh file we are going to executed the bash files to configure our mongodb containers and start the replica set.

B) CONTAINERS AND VOLUMES

Now let me talk about the pre-start.sh, in this file we are setting bash functions that will configure the mongodb docker volume, with the help of a temporary mongo container.

why do we need to this pre-configurations ?

Since we don’t have any base docker image created with all this pre-configurations, we need to do this pre-steps and basically this is to configure the filesystem of our docker volume; in the temporary container we are going to create the folders “/data/keyfile” “/data/admin” that will be stored in the docker volume, and then set the correct ownership to the folders with the following: “chown -R mongodb:mongodb /data”, also we are copying over the mongo-keyfile this to authenticate our servers to be part of the replica set, and the admin.js file to create the admin and replica users. Once this volume is already setup, then we are going to delete this temporary container, so we can create our final container. So this is how the pre-start.sh file looks like:

Next I will talk about the start.sh file. This bash script will create the final mongodb container with all the configurations set and ready to be part of the replica set, and this container will have all the corresponding flags with its corresponding values, here is where we are calling the host.sh file that we had create earlier in the mongodb-service.sh file.

So this is how we create our mongodb infrastructure in an automated fashion way, by using all this great DevOps tools, all this bash files are executed in all the 3 instances but only 1 instance will execute the primary-db-only.sh file to init the replica set configuration.

C) MONGODB REPLICA SET

Now it’s time for the last piece, joining the 3 instances and init the replica set to have it up and running. All the magic is stored in the primary-db-only.sh file, which basically will execute mongodb commands to init the replica set, add the hosts to the cluster, but first it will create the admin and replica, and test databases users then it will import the mock data into the cinemas and bookstore databases just for fun to be able to do some queries to test our infrastructure.

Basically we will force to execute this file only in one instance and this will be only in DB1 instance as you can see in the isPrimary function, also we have some helper functions like wait_database function, this is a very helpful function because we create our instances with terraform and we don’t know who will be who, and it will start at the same time also and we don’t know when all the containers will be up and running, so by using this function it will tell us when a container is ready to be added to the replica set, and it will added only when is ready otherwise, it will be wait until the container of the host selected becomes ready, and this is how we are avoid this race condition problem.

And here is how it looks like the primary-db-only.sh file:

#STEP 3 CREATE THE INFRASTRUCTRE

We are already all setup with all the configurations and all the knowledge, now we can create the mongodb replica set with some few bash commands, so it’s time for me to explain you how what are this commands.

First lets create our mongo instance image for aws, but first we need set the environment variables in our start.sh script. Once we set this values, we can run the following command inside the Packer/ directory.

Packer/ $ bash start.sh

Once packer finishes its process then we need to run terraform commands inside the Terraform/ directory:

Terraform/ $ terraform init

Once you do the init, terraform will initialize our modules so we can make use of them, it will download the provider plugins, we should see the following output:

Initializing modules...- module.aero_bastion_network
- module.aero_bastion_instance
- module.aero_mongo_clusterInitializing provider plugins...The following providers do not have any version constraints in configuration, so the latest version was installed.To prevent automatic upgrades to new major versions that may contain breaking changes, it is recommended to add version = "..." constraints to the corresponding provider blocks in configuration, with the constraint strings suggested below.* provider.aws: version = "~> 2.4"* provider.null: version = "~> 2.1"* provider.template: version = "~> 2.1"

Terraform uses a plugin based architecture to support the numerous infrastructure and service providers available. As of Terraform version 0.10.0, each “Provider” is its own encapsulated binary distributed separately from Terraform itself. The terraform init command will automatically download and install any Provider binary for the providers in use within the configuration.
- Terraform

Then we are going to execute “terraform plan”, the output of the execution plan, it will describe which actions Terraform will take in order to change real infrastructure to match the configuration. The output format is similar to the diff format generated by tools such as Git. The output has a + next to aws_instance.example, meaning that Terraform will create this resource. Beneath that, it shows the attributes that will be set. When the value displayed is <computed>, it means that the value won’t be known until the resource is created, now we need to execute the following:

Terraform/ $ terraform plan

Next we need to execute the following command to tell terraform to create our infrastructure in aws.

Terraform/ $ terraform apply

You will be prompted to set the some values, because we don’t have any default values in our variables.tf file in our root module, this is how we set the value dynamically to our configuration from the command line, we are doing this because if we commit our terraform code, we don’t want to push our aws credentials to a public or private repository, so after you do the apply you should se something like this:

var.access_key
Enter a value:var.aws_account
specify your aws number accoutEnter a value:var.secret_keyEnter a value:var.ssh_key_name
The name of an EC2 Key Pair that can be used to SSH to the EC2 Instances in this clusterEnter a value:

Once you introduce the values, terraform will output the resources that is going to create, you should see something like the following:

Then it will ask you if you want to perform the creation of the resources identified by terraform, then you will need to type yes, so terraform can start creating the resources.

If `terraform apply` failed with an error, read the error message and fix the error that occurred. At this stage, it is likely to be a syntax error in the configuration.

Executing the apply will take a few minutes since Terraform waits for the resources to become available, you should see something like the following output:

After this, Terraform is all done! You can go to the AWS EC2 console to see the created resources. (Make sure you’re looking at the same region that was configured in the provider configuration). If you are new to terraform I highly recommend you to visit and read this tutorial from Hashicorp: https://learn.hashicorp.com/terraform/getting-started/build

Now that you are at your AWS EC2 console, look for the bastionVM instance copy the public ip and get back to the terminal and ssh into your bastion host using your ec2 key pair. Once inside your bastion host, you will need to create a file like ec2.pem and it should have the same content as your key that you used to ssh into your bastion host then you will need to chmod 400 the ec2.pem to give the right permissions to file, once you done that you will need to ssh again but this time from the bastion host to one of the mongodb instance so, you can look at your AWS EC2 console again to get the ip of one of the mongo instances.

Inside of one of the mongo instances run the following command, to verify if there is the container running and should see something like the following:

if is not running is because something went wrong during the configuration process, then check what went wrong in the following log file: /var/log/user-data.log

If the container is up and running, we can double check if the replica set was configured correctly by running the following command:

$ docker exec -i $DB_HOST bash -c 'mongo -u '$DB_REPLICA_ADMIN' -p '$DB_REPLICA_ADMIN_PASS' --eval "rs.status()" --authenticationDatabase "admin"'

And then we should see something like the following output:

With the above output we have finally finished the configuration of the creation and configuration of the mongodb replica set, all created with tons of automation and with the greatest tools for DevOps such as Packer, Terraform and Ansible, I hope you have learned good content in this article about creating and provision infrastructure.

For the curious ones we can run some queries to the bookstore and cinemas databases using the following commands.

First we are going to connect to the cinemas database using the following command:

$ docker exec -it $DB_HOST mongo cinemas -u $CINEMA_DBUSER -p $CINEMA_DBPASS --authenticationDatabase "admin"

Then we should be inside the mongo shell and it looks something like this:

MongoDB shell version v4.0.8connecting to: mongodb://127.0.0.1:27017/cinemas?authSource=admin&gssapiServiceName=mongodbImplicit session: session { "id" : UUID("7f55a836-57ca-4136-9b8e-9252eb1c3575") }MongoDB server version: 4.0.8rs1:PRIMARY>

Now we are ready to query the cinemas database, so lets query the cities document like the following:

rs1:PRIMARY> db.cities.find(){ "_id" : ObjectId("588ababf2d029a6d15d0b5bf"), "name" : "Morelia", "state_id" : "588aba4d2d029a6d15d0b5ba", "cinemas" : [ "588ac3a02d029a6d15d0b5c4", "588ac3a02d029a6d15d0b5c5" ] }{ "_id" : ObjectId("588ababf2d029a6d15d0b5c0"), "name" : "Queretaro", "state_id" : "588aba4d2d029a6d15d0b5bb", "cinemas" : [ "588ac7a12d029a6d15d0b5cb" ] }{ "_id" : ObjectId("588ababf2d029a6d15d0b5c1"), "name" : "Medellin", "state_id" : "588aba4d2d029a6d15d0b5bd", "cinemas" : [ "588ac7332d029a6d15d0b5ca" ] }{ "_id" : ObjectId("588ababf2d029a6d15d0b5c2"), "name" : "Bogota", "state_id" : "588aba4d2d029a6d15d0b5bc", "cinemas" : [ "588ac65f2d029a6d15d0b5c8", "588ac65f2d029a6d15d0b5c9" ] }{ "_id" : ObjectId("588ababf2d029a6d15d0b5c3"), "name" : "Santiago de Chile", "state_id" : "588aba4d2d029a6d15d0b5be", "cinemas" : [ "588ac53c2d029a6d15d0b5c6", "588ac53c2d029a6d15d0b5c7" ] }rs1:PRIMARY>

And we can do the same steps for the bookstore database, but using the bookstore credentials.

root@ip-10-0-3-175:/home/ubuntu# docker exec -it $DB_HOST mongo bookstore -u $BOOKSTORE_DBUSER -p $BOOKSTORE_DBPASS --authenticationDatabase "admin"MongoDB shell version v4.0.8connecting to: mongodb://127.0.0.1:27017/bookstore?authSource=admin&gssapiServiceName=mongodbImplicit session: session { "id" : UUID("9670221c-1e23-4073-ad9b-cbdebb791b88") }MongoDB server version: 4.0.8rs1:PRIMARY> db.authors.find(){ "_id" : ObjectId("58b603b1ef9f298bd025397a"), "name" : "Cristian Ramirez", “occupation" : “engineer", "description" : "lorem ipsum" }rs1:PRIMARY>

Now lets destroy the infrastructure that we have created, just by executing the following command:

Terraform/ $ terraform destroy

And this will destroy and remove all the resources that we have created in aws, so we don’t get charged by aws for infrastructure that we no longer are going to use. If we want to reuse this infrastructure, you just have learned how to create it just by executing terraform commands.

TIME FOR A RECAP

What we have done …
Uff lots of information and very very good technical stuff

We have learned about the benefits of immutable infrastructure and why we should use it.

We have made our base instance image with Packer provisioning with Ansible.

We have created our infrastructure using Terraform with the AWS Provider, so we have learned what is infrastructure as code, and how to implement it.

We have seen bash scripting, automating the creation of a mongodb replica set.

We have played with Docker containers and Docker Volumes.

We have automated all the processes, from instance image creation to having a mongo replica set on containers up and running, ready to be used.

We have involved many state of the art technologies, with DevOps Paradigm and best practices plus using the benefits of the cloud and have achieved the following architecture just by using infrastructure as code.

We also have learned what we can do with aws cloud, for people who has not yet begin working with the cloud.

CLOUD ADOPTION — THE REALITIES OF THE MULTI CLOUD

Organizations of all sizes are adopting the cloud operating model for their application workloads. Whether to optimize the costs of running and managing their data centers or to enable development teams to efficiently build the applications of tomorrow, the majority of organizations are interested in the benefits of the cloud model.

But the primary challenge of cloud adoption is that all four core constituents in IT — operations, security, networking, and development teams — must each internalize the implications of the cloud model.

The cloud model assumes that infinite quantities of infrastructure will be provisioned on-demand by many teams and then destroyed when no longer required. We can achieve this by using terraform as I show earlier in the exercise, we created and provision infrastructure on-demand and we destroy it when we no longer need it.

Since cloud providers expose their services using APIs, operations teams are adopting the notion of infrastructure as code. In this manner, an infrastructure topology can be codified as a simple script and then executed each time that infrastructure is required. And as we saw in our exercise we just create a whole new infrastructure in the cloud just by creating terraform scripts.

An infrastructure as code approach enables reusability, but the reality is that large organizations will ultimately utilize more than one cloud provider in addition to their own private on-premises data center. And I can talk about it, where I am currently working, we still have our own private on-premise infrastructure where we are deploying our microservices, but have all dev environments in the cloud, but to have an immutable infrastructure we have all instances running the same instance image, and this is created with packer, so development clusters, cloud clusters and bare metal clusters have same version of image instance, but we are slowly moving to the cloud, and in our cloud infrastructure we are using more than one cloud provider, for development cluster we use aws, and we have 1 production datacenter on gcp.

Because each infrastructure provider has a unique provisioning model, a challenge for operations teams is to determine a strategy that gives them a consistent provisioning workflow regardless of infrastructure type while leveraging the unique capabilities of each cloud provider.

The adoption of cloud is often driven by new application requirements and by developers pushing a shift to the ‘DevOps’ model. The runtime layer for their applications, however, is inevitably different from previous models: rather than deploying an artifact to a static Application Server, developers are now deploying their application artifacts (often packaged into a container) to an application scheduler atop a pool of infrastructure provisioned on demand.

Organizations can address these challenges of cloud adoption with tools that provide a consistent workflow to a single, well-scoped concern at each layer of the infrastructure stack. This focus on workflows over technologies allows underlying technologies to change while the workflow for each part of the organization does not. As a result, organizations simplify challenges related to their diversity of technology.

HashiCorp provides a suite of products that address the challenges of each constituent as they adopt cloud. Each tool addresses a focused concern for the technical and organizational challenges of infrastructure automation, so tools can be adopted one at a time or all together.

Right now we have used two of five Hashicorp tools that we are going to implement in our exercise:

Packer: Which we use to create our aws image to spin up our instances with mongodb containers, by using packer we are applying to our cloud system the model of immutable infrastructure, this provide us a single workflow to package applications for any target environment, and the greatest benefit is that we can have a better control of versioning our infrastructure.
Terraform: Which we use to codify our mongodb infrastructure by using the aws provider. By applying the infrastructure as code concept we can follow the same principles that software developers use to collaborate on code( like Github or other version control systems).

In future articles we are going to see how the other 3 products of Hashicorp fits in our MongoDB configuration in a DevOps fashion style:

Vault: With vault we are going to have a centralized approach to manage our database user credentials and database encryption keys, because it is highly available and secure way to store our most sensitive information. Vault allows teams to consume the data they need without coordinating with security teams. Security teams can change passwords, rotate credentials, and update policies without coordinating across the organization.
Nomad: With nomad we are going to deploy our mongodb containers, so that our containers can be scheduled and deployed correctly, because right now in our exercise we are creating are containers manually and we are not doing a smart placement of them, with nomad we are allowing operators and developers to avoid manual coordination of containers.
Consul: With consul we are going to make our containers to have service discovery, so we can avoid the hardcoding steps where we are calling the awscli to get the ip’s of the instances and save it into individually environment variables, so we can configure and add the host to our mongo replica set, instead we can automated all of this with the service discovery that consul provide us, Also with consul we can monitor the health of our container to ensure only healthy instances receives traffic and it notifies developers or operator of any issue. This allows development teams to avoid coordinating on ip addresses, and pushes discovery into the runtime.

CONCLUSION

Cloud native technology is clear: improved agility, increased reliability and expanded horizons for innovation, and if we combine DevOps model implemented in the right way maximizes the velocity of software delivery. The success factor of this improved workflow is the use of automation across a wide range of activities and processes in multiple areas including testing, deployment, and configuration management. But like any paradigm shift, the transition faces some serious obstacles. Efforts to adopt cloud native technologies and DevOps model face some of the hurdles you’d expect any new technology to come up against: complexity and lack of training.

The DevOps culture necessary to effectively use open source, cloud native technologies has fundamentally changed software and team processes. It is expanding how we work and think. Startups, in general, are on board. They don’t have entrenched technology that needs to be maintained and upgraded. They are also able to hire people whose skill sets are a good fit with newer technologies. For enterprises, it’s a bit tougher. They have massive investments in workhorse technologies and platforms. But they also have IT teams with deep heritage and operational knowledge in building, deploying, running and maintaining applications over decades.

The definition of DevOps varies from business to business, but the zeitgeist of DevOps is about minimizing the challenges of shipping, rapidly iterating, and securing software applications. DevOps primarily involves the people responsible for delivering applications, including developers, operators, and security professionals. These three interdependent roles need tightly coupled tools to coordinate their contributions to application delivery. The end result is a process that prioritizes agility, time to value, and a more continuous integration and continuous delivery model. When DevOps elements are effectively integrated, there will be a greater return on investment (ROI): Product time to market will decrease while quality increases.

The rise of DevOps is also tied to the rise of hybrid cloud infrastructure, characterized by distributed services and data center resources. Modern applications are Internet-connected and have thin clients such as browsers and mobile apps. Updates can be delivered quickly and there is often no “recall” that requires more disciplined risk management.

GITHUB

Crizstian/mongo-replica-hashicorp-stack

This repo is about creating a mongodb replica set using Packer, Ansible, Terraform and Aws to provision it …

github.com

Here is the full code of this repository please use v0.1 since master branch will be changing for the further articles. Happy Coding, feel free to make suggestions or contributions to the code, i will love to see what other people does.

REFERENCES

How to deploy a MongoDB Replica Set using Docker
Docker documentation
Terraform documentation
Packer documentation
Mongodb documentation
AWS documentation
Valaxy AWS VPC tutorial (YouTube)

What’s the biggest thing you’re struggling with right now in DevOps that I can help you with?

What kinds of content would you like to see more on this new article series about DevOps?

I hope you enjoyed this article; I would like to know your comments; If you enjoyed this article, please refer it to a friend, collogue, etc.

Until next time 😁👨🏼‍🎨👨🏻‍💻

Get entire infrastructure defined as code in about a day and get it fully implemented, tested and versioned infrastructure on AWS or any other cloud. Feel free to reach me at LinkedIn.

About Me

I am a DevOps Engineer / Software Engineer with over 7 years of experience. I have experience on implementing a container based CI/CD/CM Pipeline for cloud and hybrid environments. Expert on Hashicorp Tools and Golang Development. Also I Had successfully lead efforts to introduce and evangelize microservices development and deployments, Rich exposure to the container orchestration ecosystem. Follow me on Twitter at @cramirez_92 and on LinkedIn.

Favorite Quote
“You can compete or you can dominate. One makes you better the other makes you the best”
– Grant Cardone