Microservices 101 (and building one using Docker, Swagger, Knex and Express)

And no, you probably don’t need that fancy microservice architecture you’ve been eyeing.

Before we start and for the record, the answer is no. You very likely do not need to implement a microservice architecture for your new idea/latest project/startup that’s going to change the world with an elevator pitch (I hear it’s now known as watercooler pitch) of sorts.

So why a walkthrough on microservices?

Because we can. Got a problem with that?

Because we can; and also because it helps in later stages of application development and operations that demand predictability, reliability and ease of targetted scaling to consistently achieve the right level of availability, but as geeks, we all know that’s all secondary… 😅

👀 TL;DR

• Start with Why
• If not now, When?
• Finally How: What Goes Into A Microservice
• Hands-On: A Microservice At The Application Layer

🤔 Start with Why

Web applications have traditionally been monolithic. The better among the good adopted a Service Oriented Architecture (SOA), which was essentially still a monolith, but separated into various services that a central controller coordinated.

The microservice architecture, as compared to SOA, is composed of infrastructurally independent services communicating with each other via pre-established protocols like SOAP/WSDL/REST on XML/JSON.

So with that, here’s some reasons you might want to consider a microservice architecture:

1. Never before have we had so many people online, with many more to come, necessitating the invention of new means to maintain availability of services. At roughly 47% of the world population connected to the internet in 2017 as compared to about 8% in 2001 (source: ITU Statistics 2016), the average number of hits per website can only be expected to increase.
2. Monoliths serve as an excellent single point of failure. Monolithic code is great for new businesses/products, enforcing convention over configuration and expediting development by aligning a possibly freshly put together team on a set of recommended best practices. However, as software grows, so does it’s complexity, along with the consequential likelihood of bugs. All it takes is one bug in one module to cause the world to burn. The microservices architecture (assuming no arrangements for high availability at the infrastructure level) is one way of improving availability by avoiding a single point of failure within your application, lessening the likelihood of all your services being down at any given point in time.
3. Development bottlenecks will start to appear in the form of insufficient test runners and merge conflicts with shared files as teams grow. As software grows, so do the teams behind them; and as the team size grows, commits and pushes become more frequent, resulting in symptoms such as CI pipelines requiring longer durations to complete and more frequent conflicts in common files. Microservices remove this pain by allowing for cross-functional teams to work on different features completely independently on different repositories.
4. The microservices architecture aligns with current best practices in management of development teams. As software houses increasingly shift from Waterfall to Agile development methodologies, teams necessarily become cross-functional, resulting in units of developers/designers who are able to create end-to-end functionalities as a unit. Microservices fit in by allowing for the organisational evolution of cross-functional teams around feature sets.
5. Being independently deployable, microservices allow for teams with varied knowledge across multiple technologies to operate on the same product, bringing the strengths of multiple technologies into one product. Think being able to combine Elixir’s concurrency, Node’s event driven architecture, and Rust’s reliability within your product. Not so possible if your product is based off Ruby on Rails (which is an awesome framework in its own right).
6. With microservices, we get to apply targetted scaling according to which functionalities are most in demand. When using monoliths, scaling is done on an all-or-nothing basis — the application either scales as a whole or it doesn’t — disregarding the actual performance bottlenecks. Breaking the monolith into loosely coupled microservices that are properly monitored, allows for the scaling of just the service that requires more/less computing power. In the spirit of applying technology where applicable (see point 5), we could even do a rewrite of the offending service in a more suitable language/runtime/framework that handles the problem more efficiently. That won’t be possible with a monolithic architecture.

⏰️ If not now, When?

However, that being said, implementing a microservice architecture is over-engineering at it’s finest when the need is not there. What constitutes need is subjective, but some key indicators that your application is ready to/should evolve into a microservice architecture are:

1. Multifaceted application that solves disparate business problems (ie not for your startup idea that does one and only one thing extremely well).
2. High user traffic patterns/drastic usage spikes.
3. Heavily skewed utilisation of certain features.
4. Performance bottlenecks that could be easily solved by using a different technology.
5. Complex codebase with increasing rates of modifications to central configuration files increasingly causing merge conflicts.
6. Constantly overloaded Continuous Integration/Delivery (CI/CD) pipeline even after sensible levels of scaling.
7. Growing number of cross-functional teams who do development in specific business features.
8. Availability of engineers with varied expertise covering multiple technologies.
9. Availability of engineers also competent in DevOps/Ops.

🎁 Followed by How: What Goes Into A Microservice

A microservice is an independently deployable application that respects the Single Responsibility Principle above all — it should handle one aspect of the application, and handle it extremely well.

Inspired by distributed object oriented programming, the microservice architecture concerns itself with a single business-level feature. For example in an internet application that manages events for organisers, there might exist an accounts microservice that handles all operations related to user accounts, another events microservice that handles operations related to events, possibly a payments microservice that handles just the payments.

Being self-contained means that each microservice will need to encapsulate it’s own:

1. Operating System—what base system should we run on to support our required runtimes/frameworks?
2. Runtimes/Libraries/Frameworks — what software should the base system come installed with to support our application?
3. Database Interface — what persistent storage mechanism are we using?
4. Application — what logic does this service encapsulate?
5. Monitoring/Scaling Systems — how can we know more about its performance and increase/decrease number of instances accordingly?

Containerisation technologies such as Docker settle the availability of (1), (2), and (3) for us through images from it’s registry, leaving us to handle (4) — ie code, the fun part — which we will write in Node.js for the hands-on section below. Lastly, (5) is pretty much out of the scope of this post, but of utmost importance to successfully adopting and maintaining of a microservices architecture, so there’ll be some links to additional resources at the end if you’re interested in digging.

Additionally, we need to implement microservices to provide for (adapted from Martin Fowler’s Microservice Pre-Requisites):

• Rapid provisioning
• Rapid deployment
• Constant monitoring

All of which are essential to reap the benefits of ease of targetted scaling and maintaining of availability. A more extensive & verbose list of good practices for applications built according to the service-oriented architecture that also applies to microservices can be found at The Twelve-Factor App which in summary is as follows:

1. Codebase — One codebase tracked in revision control, many deploys
2. Dependencies — Explicitly declare and isolate dependencies
3. Config — Store config in the environment
4. Backing Services — Treat backing services as attached resources
5. Build, Release, Run — Strictly separate build and run stages
6. Processes — Execute the app as one or more stateless processes
7. Port Binding — Export services via port binding
8. Concurrency — Scale out via the process model
9. Disposability — Maximize robustness with fast startup and graceful shutdown
10. Development/Production Parity — Keep development, staging, and production as similar as possible
11. Logs — Treat logs as event streams
12. Admin Processes — Run admin/management tasks as one-off processes

🔌 The Shameless Halfway Plug

For the less geeky amongst you, this article has served it’s purpose.What follows is a walkthrough on how to get up a containerised microservice using Docker for containerisation, Swagger for API management, Knex for database management, and Express as a server.

If you’ve found this a nice read so far, consider clicking on the 💚 (⬇️ for mobile, ⬅️ for desktops) to improve the visibility of this post so that Medium will share this with more people like yourself and they can benefit as well(: you can follow me/our GDS blog too to receive updates when I/we make posts like this. Cheers! 🍻

With that, should you choose to continue, grab a cuppa and your fanciest mechanical keyboard and… I’ll be here waiting.

🙌🏽 Hands-On: A Microservice At The Application Layer

Enough business talk. We begin here by preparing a simple API service that’s independently deployable across any environment. Note that this covers just the application level architecture and does not include peripheral tools such as monitoring, auto-scaling and storage volume management which are essential to the microservice architecture. That’s another story for another post! So,

👀 TL;DR

• Pre-Requisite: Install Docker
• Pre-Requisite: Install Swagger
• Initialise your Application
• Define the Application Container with Dockerfile
• Define the MySQL Container with Docker Compose
• Define the Docker Entrypoint
• Initialise Knex
• Test Drive the Container
• Create the Migrations & Seeds

🐋 Pre-Requisite: Install Docker

Released in 2013, Docker remains the only one of it’s kind that’s open-sourced. We use Docker to containerise our application, providing it with the required runtime (Node) and database (MySQL).

Download the Community Edition from this link for your platform of choice and install it.

😎 Pre-Requisite: Install Swagger

Swagger began as an API documentation tool, spawning the OpenAPI standard. As it grew, Swagger gained the ability to generate boilerplate code via swagger-codegen, but that’s another story for another day. We use Swagger together with it’s Node adapter to quickly generate a boilerplate for our API.

We install Swagger globally via npm:

> npm install -g swagger

🤓 Initialise your Application

We begin by using Swagger to initialise our app. Go to a folder of your choice and run the following which will create a directory called microservice in that folder:

> swagger project create microservice

When presented with the Framework? question, select express and wait for the dependencies to install. When done, you can cd into the folder and run npm start. All should be well and you’ll get a server accessible at localhost:10101. Kill it quick with ctrl+c, before it becomes another monolith. I jest.

While we’re at Node, let’s also set up the rest of our dependencies, namely Knex, MySQL connectors and some other peripheral packages. In your project root, run the following:

yarn add knex mysql2 body-parser helmet

Also, open the package.json file and search for the scripts hash property. Add a script as follows:

{
  ...,
  "scripts": {
    "development": "nodemon app.js",
    ...
  },
  ...
}

This will allow us to do npm run development to get our server up using the nodemon process manager which we will install globally via the Dockerfile. If you’ve done development in Node before, don’t worry about the NODE_ENV either, we’ll settle that using Docker.

📦 Define the Application Container with Dockerfile

For the sake of learning to build our own image, we avoid using the node image from the Docker registry but you could try it out by yourself later. Create a file named (surprise surprise) Dockerfile in the root of the project and paste in the following code:

Let’s run through it line by line. Code precedes the description.

FROM centos:centos7

This specifies the base image we want to use. In this case, we’re grabbing a CentOS 7 image.

RUN yum -y update

This updates the yum (Yellowdog Updater Modified — if you’re interested) package manager’s repositories, making sure we have an updated list of all available software. The -y indicates that we wish to say yes to every question yum throws at us.

RUN curl --silent --location https://rpm.nodesource.com/setup_7.x | bash -

We proceed to set up Node 7 by first downloading, installing and enabling the RPM (RedHat Package Manager) file for Node.

RUN yum -y install gcc gcc-c++ make nodejs

Next we install the possible dependencies. gcc, gcc-c++, and make, which are required by some Node dependencies to build native binaries.

RUN npm install -g yarn nodemon

Finally, we install a better package manager (sorry, npm), yarn, and nodemon. nodemon allows us to have a nice development environment where server code is live reloaded as changes are made.

That wasn’t so tough was it? Now that we’ve got ourselves a nice container specification, why not build it? Run the following command in the root of your project:

> docker build .

Don’t forget the trailing period. You should then see a series of Step x/5 line prints that represent each line in your Dockerfile. When it’s done, a Docker image would have been built. Verify this by running:

> docker image ls

You should see an image that was recently built with centos as it’s repository and centos7 as it’s tag.

🗄 Define the MySQL Container with Docker Compose

Next, we need to define a database we can use to provide data persistence to our application. We could use another Dockerfile, but since we’re exploring Docker, let’s use docker-compose, which is a smarter way to do things when two or more services need to be run at the same time.

Create a new file named docker-compose.development.yml. This will be the development Docker configuration. You can go on to create qa, uat, staging and production as your application scales. Paste the following inside:

You can see two services being defined there, app_dev and db_dev. Numbers here might seem weird if you’ve done development in MySQL before, but we shall use weird numbers to show how we can alter defaults while proving that the configuration was indeed applied.

Let’s go through app_dev first. As before, code precedes the description.

build: ./

This tells Docker where to find a Dockerfile (the file we created in the previous step).

container_name: app_dev

This names your container and allows for easier management of containers when the number of containers being managed grows large. The container_name property also serves as the hostname for each containerised service when you run docker-compose.

entrypoint: ./docker-entrypoint.sh

This defines a file to run upon successful startup of the container. We will create this file later.

environment:
  - NODE_ENV=development
  - DB_RESET=true
  - DB_MIGRATE=true
  - DB_SEED=true

This specifies environment variables not unlike your .env file of yore. Since we are defining our development container, we set the NODE_ENV to development. The other three environment variables will tell our application whether to reset our database schema, perform migrations, or seed the database, all of which we might want to toggle on/off in different environments in future. For our development container, we set everything to true so that we have a reliable environment to build on.

ports:
  - "3000:3000"

This indicates that we wish to map the internal port 3000 (the latter 3000) to our host’s port 3000 (the former 3000).

volumes:
  - ./:/var/www/app

The volumes property lets us define a shared folder of sorts between our host and the guest. Here we indicate that our current project folder should be shared with the directory located at /var/www/app within the container.

working_dir: /var/www/app

The working_dir property tells Docker where in the container we should initialise the application in.

Whew. Now for db_dev, but you should already be getting the hang of this. I’ll cover only properties not already covered from app_dev.

image: mysql:5.7.18

Here we specify an image, mysql:5.7.18, which is the latest MySQL General Availability at time of publication. This tells Docker to download the image from the Docker registry. Note that this tells us nothing about the infrastructure, and leaves it to the image maintainer to decide the best for us. It could be fedora or even centos. We don’t care about such trivial details anyway. The Dockerfile we defined earlier is an example of such an image others could use for a Node setup if we were to put it on the Docker registry.

command: mysqld --user=root

This tells Docker the command to run upon successful booting of the container. We initialise the mysqld daemon using the root user here.

volumes:
  - ./provisioning/mysql/development:/etc/mysql/conf.d

We’ve already covered volumes, but in this case, it’s necessary to understand how to configure images. In the case of the mysql:5.7.18 Docker image, the settings are located at /etc/mysql/conf.d which we map a local folder to so that we can inject our own settings. We will create the required files later.

expose:
  - "3304"
ports:
  - "3303:3304"

The expose property tells Docker to expose port 3304 over the internal network Docker creates to link our defined services. The ports property we have already covered. In this case, we are mapping port 3304 from the guest (container) to port 3303 of our host (your laptop). This is necessary so that we can run in development, staging, production, et cetera all at once by mapping their databases to different ports on our local machine. As explained above, numbers here are intentionally set to non-defaults, you really should use 3306 on the guest for actual development and map it to 3303 on your host.

Next, create the configuration for our mysql service by creating the directory at ./provisioning/mysql/development:

> mkdir -p ./provisioning/mysql/development
> cd ./provisioning/mysql/development

Create a new file named mysql.cnf inside the folder and paste the following into it:

[client]
port=3304
[mysqld]
port=3304

This tells mysql to listen on port 3304 instead of the default 3306. The .cnf extension is important because that’s how MySQL knows which files to load.

Awesome, we’ve set up our Docker configurations! But wait, what about docker-entrypoint and friends?

🚩 Define the Docker Entrypoint

Now we give Docker a script to run for our application (app_dev). Let’s create the docker-entrypoint.sh file in the root of our project and add the following to it:

The code should be self-explanatory. We wait for MySQL to be ready for connections, checking for connectivity every one second. Then we proceed by checking for the DB_RESET, DB_MIGRATE, and DB_SEED flags to see if we should reset/migrate/seed the database respectively.

Since this file is going to be run via the shell within our container, we need to give execute permissions to it. Give it the permissions it needs with:

> chmod +x docker-entrypoint.sh

From the code, we obviously have another to create, ./provisioning/wait-for-mysql.js. Create that file and paste the following into it:

And just one last file to provide the database configuration for our app. Paste the following code into a file at ./config/database.json relative to the project root:

📂 Initialise Knex

Now we initialise our database management tool, Knex, using the following command. Run it from your project root:

> ./node_modules/.bin/knex init

A knexfile.js should have been created for you. Let’s change the settings to suit what we have defined above in our Docker configuration. Override (paste over) the pre-defined environments using the following lines with:

This indicates we will be using the mysql2 client which we’ve installed earlier together with the development database configuration. We have also indicated a migrations and seeds directory, let’s go ahead and create the new folders:

> mkdir -p ./migrations/development
> mkdir -p ./seeds/development

⛷ Test Drive the Container

At this point, we are done with the Dockerization and setting up and you can now run the following command and experience your (first-ever?) container in it’s full glory:

> docker-compose -f docker-compose.development.yml up --build

You should see a build process similar to when you ran docker build . but this time, it should also start the container because of the up parameter. To explain some things in the output:

Building app_dev

Below this line you should see the same output as when you ran docker build . earlier. This is because we have specified in our Dockerfile in the build property to use ./ which indicates to Docker to use the Dockerfile found at ./.

Pulling db_dev (mysql:5.7.18)...

This is where the image we defined in the db_dev service is being pulled from the Docker registry.

Creating db_dev ...
Creating app_dev ...

Containers are being created here using the images we built (for app_dev) or downloaded (for mysql:5.7.18). You should see app_dev and db_dev appear in contrasting colours on the left. This is the output from each container. You should see our wait-for-mysql.js script waiting for the port 3304 before executing the rest.

To confirm that we have got the configuration for MySQL right, check for a line with the following:

Version: '5.7.18'  socket: '/var/run/mysqld/mysqld.sock'  port: 3304  MySQL Community Server (GPL)

If you’re using a different version of MySQL, replace '5.7.18' with the version. The port: 3304 indcates we are listening on port 3304 inside the container and the internal Docker network but doesn’t tell us whether we can access it via port 3304 from our local computer. Let’s check our local ports with:

> lsof -i :3304

Nothing!

Let’s check port 3303 which we have also defined earlier inside the ports property inside docker-compose.development.yml:

> lsof -i :3303

You should be able to see a vpnkit command running on that port. vpnkit is what helps to forward the internal network’s port to our host computer’s port. We should be able to connect to our database using the following command:

> mysql -P3303 -h127.0.0.1 -uroot -ppassword

📩 Create the Migrations & Seeds

So we’ve defined DB_RESET, DB_MIGRATE, and DB_SEED in our docker-compose.development.yml, let’s put these to use.

Create a migration by running a Knex command in the root of your project:

> ./node_modules/.bin/knex migrate:make create_users

Open the folder located at ./migrations/development and you should see a file named YYYYMMDDHHmmss_create_users.js. This is the migration we just created. Let’s add the details for the migration; copy and paste the following into the YYYYMMDDHHmmss_create_users.js file:

We’ve indicated that the columns: id, email, password, andlast_login should be defined together with the normal timestamps, and we define it inside exports.up which is run when creating the table users. We also specified exports.down which let’s us undo the migration if need be.

Next, let’s seed our database with some basic information like an admin account. Run the following command to create a Knex seed file:

> ./node_modules/.bin/knex seed:make add_admins

Copy and paste the following code into the file at ./seeds/development/add_admins.js:

These 3 users will be inserted on starting our Docker container. Now we’ve got the migrations and seeds up, we can run our containers to watch our database get migrated and seeded by running docker-compose up:

> docker-compose -f docker-compose.development.yml up --build

You should be able to observe the following lines:

...
app_dev    | * Running Database Migrations *
app_dev    | Using environment: development
app_dev    | Batch 1 run: 1 migrations
...

This confirms that the migrations were run. You should also be able to observe the following lines:

app_dev    | * Seeding Database *
app_dev    | Using environment: development
app_dev    | Ran 1 seed files
app_dev    | /var/www/app/seeds/development/add_admins.js

This confirms that the seeds were run. If you’d look into your database right now, you’d be able to confirm the users table with 3 entries.

Whew. That wasn’t too bad wasn’t it?

🚲 Moving Forward

Create the Application

We now have an infrastructure to continue creating a microservice. Next steps would be to modify the API to provide for:

1. User registration
2. User authentication
3. Account management

Which together, would evolve what we have made so far into a full fledged users/account microservice we can use for any application that requires remote authentications.

Choose an Operating System

Despite Docker being able to normalise most infrastructure/network level differences, Docker still needs to run on a base kernel which can be CentOS, Ubuntu or even Windows (but why?). There are systems made with hosting of containers in mind. One of these is CoreOS.

CoreOS

Alternatively, pull an image from Docker Hub at https://hub.docker.com/.

Deploy & Make the Application Scalable

Implementing a microservice architecture is useless if we are not going to allow it to be scalable. We can do this by using available tools like Docker Swarm which comes from the Docker team:

Swarm mode overview

There’s also startups like Puppet dealing with the problem:

Docker

Or Kubernetes if you’re looking for a more enterprise level solution that’s also backed by Google:

Kubernetes

Set up Monitoring Facilities

Without having statistics on the performance of your microservices, it is impossible to effectively perform scaling. Having multiple fragments of your application being independently fallible demands that you are able to know when any of them are down. Some tools that seem promising are:

Modern monitoring & analytics

Dynatrace Application Monitoring (AppMon)

Prometheus - Monitoring system & time series database

Cheers

You made it through!

If you’ve found this a nice read, consider clicking on the 💚 (⬇️ for mobile, ⬅️ for desktops) to improve the visibility of this post so that Medium will share this with more people like yourself and they can benefit as well(: you can follow me/our GDS blog too to receive updates when I/we make posts like this.