How we provide value to our customers in an efficient way: CI with containers

Orchestrating our CI with Docker containers

Introduction

Welcome to the first post by Promofarma’s QA Team! In this post we are going to overview how we deploy our code to production, future posts we will break down each part, going into more detail. We have implemented this CI/CD system to reflect the philosophy of Promofarma, where adapting to the needs of our customers as quickly as possible is one of our main objectives.

Before the implementation of this system, the deploys to production were made via a script, after manual testing by the development team, business and an external service.

Strategy and goals

To be able to put the business features into production in a fast and safe way. From the point of view of the code, leaving aside the infrastructure issues which will be discussed in the future, the idea is to put in all the effort before the new code reaches our master branch so that the deploy to production is a quiet and agile process in our day to day.

To achieve this, the whole process has been automated, including the testing of the code and the feature. A container-based infrastructure has been set up that controls the entire process and pre-production environments have been created, some reserved for carrying out all the automatic processes and others open for developers and business tests. Since obviously, manual testing can never be absent.

How we setup the infrastructure

The CI/CD server (Continuous Integration / Continuous Delivery) we have assembled takes into account our experiences with the technologies used and the problems we have encountered in other projects, such as:

• Software update, for example updating the Jenkins version might cause some important plugin that we use to stop working.
• Configuration changes that need to be tested before activating, you have to stop the system or mount a parallel infrastructure to test the changes.
• Technologies that with time and use begin to consume the machine’s resources (RAM, disk, etc …) such as Selenium, which after a while running on a machine begins to run it out of RAM.

To try to avoid these kinds of problems we decided to set everything up as services within containers, which gives us the flexibility and speed we seek to apply changes and free up resources. The server is very basic, it only has the OS (Ubuntu Server) and Docker installed, which is the container technology we have chosen. The basic scheme of operation is the following:

All services that need to start other services have Docker client mounted in their container. To be able to start other services we have connected the Docker socket of the containers with the Docker socket of the machine, so that even though we are executing Docker instructions from inside the container it is really happening outside and so we are able to start any service. We share with you the instruction since it was a great discovery for us considering its simplicity:

-v /var/run/docker.sock:/var/run/docker.sock

We decided to put the following elements in volumes, because they are necessary so that other services can do their work and so that we can consult them if needed:

• All configurations that should persist over time
• All the objects generated: artifacts, screenshots from the tests, logs, etc …
• The code

To control the whole process we use Jenkins, which is the only service that is always running, the rest of services are started on demand and eliminated when they are no longer needed. In this way we are only spending resources when they are necessary and we are optimizing the performance of the infrastructure.

For each process or pipeline that starts in Jenkins that requires other services, a Docker network is created which connects them and is eliminated along with the containers once it is finished. This allows us to run acceptance tests very easily by having Selenium in the same network as the container that contains the code that executes them, and this applies to the rest of services as well.

The path from a feature to production

To say that a feature is ready to be integrated it must pass the tests in a pre-production environment, the business review and the code review.

Once we have the feature ready to be integrated, the way to do it is through a ‘pull request’ against the branch ‘master’ and this is where our CI system is launched that launches all our test packages ( unit, integration, acceptance) against the branch that was generated in the ‘pull request’:

In order to integrate our branch into master we require, in addition to the tests in green, the review of at least two developers.

Once the code is integrated into the master branch, we have two ways to deploy the code to production: we can start the deploy or it can be started automatically. At the moment we are still testing this second option, which will be used to get the features that are ready to be integrated into production automatically.

In the deploy phase we will automatically test the master branch but not with the same intensity that we did before integrating the feature into it. If a feature has been integrated into the master branch it is known to be good and so in this phase we only do a ‘sanity check’, looking at the most important and critical aspects of our platform to ensure that our users will be able to continue using it, it is not necessary to go through all the tests again. Thus we gain speed in the deploy by trusting in the previous steps.

We have also added a package of post-deploy tests which make real purchases in our platform, and returns them, for quick feedback. Obviously we also have the rollback ready in case something has escaped us.

The whole system sends Slack notifications if something goes wrong during its various phases and the final status when the process has finished. In addition, the post-deploy tests inform us of the purchases made and their returns so they can be reviewed.

Some benefits

Throughout this process we have found some topics that we think interesting to share with you in case it helps you in the future. For us, they have been important issues that have helped us to make our platform more solid.

For the integration and acceptance tests we use Selenium, but we have found that whether started in Standalone mode or in Grid, with use and time it starts to consume the RAM of the machine in which it is mounted and requires a reboot. By putting it in a container, creating it on demand and killing it when finished, we have managed to avoid this and other problems. Each time the tests are launched everything starts clean.

To test new versions of any of the software in our CI/CD system, a Docker image is generated with the versions that we are interested in, we test it on the machine without interfering with the current system and when we do want to put it into the system it is as easy as eliminating the image that was being used and tagging this new image to be used by the system.

Conclusions

The best sign that this system is helping our team is that we have increased by 5x the number of deploys per sprint. Smaller features are deployed which gives us more control over the code that goes into production. Automating the entire process of deploying code to production is a highly recommended step for any development team.