DevOps and Microservices

Krishna Varma

Published in

TechMonks

11 min readJul 14, 2019

Reading Time: 10 minutes

What is DevOps?

Agile software development helped in bridging the gap between business user, developer & tester by enabling more collaborating. It helped in cutting down the time to market and develop systems in an incremental fashion. To achieve this a key element is to bridge the silos in development and operations teams. DevOps helps in brining practices to automate processes that are historically manual, siloed and slow. Organization can create efficient and effective teams by adopting Agile, DevOps and Microservices.

It is a cross-disciplinary community of practice dedicated to the study of building, evolving and operating rapidly-changing resilient systems at scale — Jez Humble

DevOps empowers every development team to be agile and accomplish their tasks with less dependency on other teams. DevOps helps in shared responsibility between development and operations teams. One major shift the development team should adopt is Building quality into the development process. Emphasis on quality should start from planning itself. Every check-in is tested using the unit and integration testing. Unless the team follows proper white box testing the benefits of DevOps can not be seen.

There are innumerable tools to setup DevOps, but the success comes with right culture in the team

A development team partly owns the responsibility of releasing the features till production and monitor. It helps a developer understand the issues in deployment and need of effective logging. It enhances the efficiency of the developer. At the same time the operations team shares the systems business goals. Operations team involves in defining the application architecture and coding practices followed by the development team. This in turn helps the Operations team to be more effective in their role.

DevOps in Microservices

Microservices as we know are independent blocks which are stitched together to build a business process or a business application. Each microservice varies independently hence their deployment happens independently. Given this background DevOps becomes core for any microservices development team. The practices followed in DevOps help the team to quickly release the features and not impact other services which might depend on its functionality.

Docker allows you to package an application with all of its dependencies into a standardized unit for software deployment.

Containers technology is a game changer for Microservices. Each microservice packaged in a container becomes the single deployable component. Containers help the DevOps team to respond to spikes in network calls by scaling up or down. Container help in cutting the testing cycles. A container created and tested in QA environment gets promoted to other environments like Staging and production. In a way we are ruling out issues from building the code in every environment. All the configurations are maintained external to the container so that they can vary when the environments change.

Phases of DevOps maturity

Continuous integration

It is a development practices and it requires the developers to integrate the code into a shared repository very often. There is no defined frequency, but it is expected every developer would check-in multiple times in a day. Each check-in is verified against the automation tests, allowing the teams to detect problems early. Continuous Integration is the first step down the path towards DevOps maturity.

Best practices in Continuous integration

Quality first: Emphasize on writing the unit and integration tests for every component. These tests should act as gatekeepers for maintaining the quality of code. Code coverage close to 100% is desirable. In Microservices architecture the coverage for all integration points (contracts) should be 100%.
Frequent code Check-in: Team members should inculcate the habit of checking in code frequently. It helps in identifying the issues early in the development cycle
Automate build: Build process should be automated. I am sure everyone these days follow this aspect. It is important to note the build time should be very minimal. It goes well with Microservices since each service is small and independent.
Communicate: This is a very important aspect of DevOps and it should happen through tools. Transparent communication across the stages of code check-in, build on integration box, execution of unit & integration tests, deployment/rollback is key. Penalize the team that contributes to more build failures.
Code smell: Measure the code quality through the linting tools available. We could use code dashboards like Sonar Qube to measure the code quality. Build automation should have a step included to push the code metrics into sonar Qube. By giving importance to code quality issues development manager can create right team culture. These tools measure the technical debt and cyclomatic complexity of the code.

Continuous integration helps in faster feedback, increased transparency, reduces integration time, improves build quality, enables testers to focus more on business features.

Continuous Delivery

It is the next step in the maturity model for DevOps. Continuous integration will help to delivery effective builds and automate the testing as much as possible. To adopt continuous delivery, we should follow all the practices of continuous integration and on top of it automate the release process. The trigger to release should be manual but the actual release process is automated.

To successfully implement continuous delivery, you need to change the culture of how an entire organization views software development efforts.

Continuous delivery can be achieved by using tools like Jenkins, CircleCI, Bamboo. Jenkins is opensource and widely used for establishing delivery pipelines. Every change to your code goes through a complex process on its way to being released. This process involves building the software in a reliable and repeatable manner, as well as progressing the built through multiple stages of testing and deployment. Delivery pipeline is a workflow of different stages in a release process. Jenkins can be configured to take an action based on the result in each stage. Pipelines in Jenkins can be defined using declarative pipeline syntax on Jenkinsfile.

Best practices in Continuous Delivery

API versioning: Teams should follow API versioning. Any changes to the existing features should ensure backward compatibility.
Feature flags: Design to place the new features behind feature flags. Continuous check-ins result in pushing incomplete features into production. Feature flags will help the product owners to enable the feature when it is ready. Product owners will have the flexibility to selectively enable the features to few early adopters and seek feedback. It helps to access the customer response on new features.
Effective planning: Particularly in Microservices architecture the services and business application might be developed by different teams. Develop API’s first and then build the business features consuming the API’s. Planning and communication across these Scrum masters is key.

When the team reaches to this maturity they can make a release anytime they want. Releases can be daily or weekly or what every suite the business requirements. However, if a build is ready and waiting for deployment to production then the essence is lost. Unless there is a compelling reason production deployment should happen without any delay.

Continuous Deployment

This is the highest level of DevOps practice. The release process goes one further step and deploys code into production without manual intervention. When a developer check-in code, it passes though the quality gatekeepers like Unit tests, integration tests, Acceptance tests. Production release happens only when the build successfully passes through the quality gates. This practice needs extreme level of maturity in the team and usually the team following this approach are experienced and self-driven.

Branching strategy

Distributed version control systems give team members flexibility in using the version control to share and manage code. Adopting a branching strategy so that you collaborate better and spend less time managing the version control and more time developing code.

Feature branches

The philosophy behind feature branch workflow is that all feature development should take place in a dedicated branch instead of master branch. It helps the developers to maintain the changes related to a feature in a branch until it is complete. It will help the master branch to be clean and ready for doing other releases. If more than one engineer works on a feature, all of them should work on the same feature branch. The other option is every developer branch out of feature branch and check in to their individual branch. Encapsulate a feature behind a feature flag.

Feature flags add technical debt over longer period. Team should clean up feature flags once the feature is stable in production. Follow proper naming while creating the feature branch. Feature branches should be cleaned up.

It is suggested to create the feature branch out of development branch and not master.

Master branch

Master branch usually contains the deployable code into production. The master branch is meant to be stable, ever push to master is thoroughly tested and passed through quality gatekeepers. Do not ever check-in code into master directly. Follow a through review of the pull requests into master. The review that takes place in a pull request is critical for improving code quality. Only merge branches through pull requests that pass your review process. Avoid merging branches to the master branch without a pull request.

Release branches

Some teams follow release branches. They give flexibility to stabilize the features and do any bug fixing while the next set of features are parallelly developed on other branches. Create a release branch from the master branch as you get close to your release or other milestone, such as the end of a sprint. Give this branch a clear name associating it with the release. Create branches to fix bugs from the release branch and merge them back into the release branch in a pull request. Code from release branch should be merged into Master as well as do a back merge into dev and feature branches.

In my experience it is better to avoid the release branches as much as possible.

Infrastructure as Code

The idea behind Infrastructure as a code is to automate the IT and release process through code. Prevent manual intervention by driving the provision, deploy, test and decommission the compute and storage resources through code.

Infrastructure as code uses higher-level or descriptive language to code more versatile and adaptive provisioning and deployment processes. For example, infrastructure-as-code capabilities included with Ansible, an IT management and configuration tool, can install MySQL server, verify that MySQL is running properly, create a user account and password, set up a new database and remove unneeded databases.

Kubernetes I the most widely used orchestration platform for running and managing containers at scale. Your infrastructure is written as a code using Kubernetes. It can be versioned and easily replicated elsewhere. By versioning the infrastructure, we could look at how the infrastructure has varied over a period.

Kubernetes groups containers that make up an application into logical units for easy management and discovery. Kubernetes builds upon 15 years of experience of running production workloads at Google, combined with best-of-breed ideas and practices from the community.

Cloud native with Infrastructure as code helps to dynamically scale the infrastructure based on the user traffic. Infrastructure as code along with self-healing is a game changes in managing the production deployments.

DevOps stack

Following tables captures some of the options available in the market for each category. These options are from my knowledge and experience hence I could have missed a popular one

CI tool: Jenkins, GitLab CI, CircleCI, Travis CI, TeamCity, Bamboo, VSTS

Source control: Github, SVN, Mercurial, Perforce, TFS, Bazaar,

Unit Testing: NUnit, Junit, TestNG, RSpec, xUnit.net

UI Automation: Selenium webdriver, TestComplete, Ranorex, CodedUI, UFT

API automation: Postman, SoapUI, Swagger

Performance testing: JMeter

Dashboard: SonarQube, VSTS

Docker Registry: Docker Hub, AWS ECR, Google Container Registry

Infra as code: Kubernetes, Chef, Ansible, Puppet

Cloud providers: AWS, Microsoft Azure, Google Cloud Platform

Application Monitoring: ELK, ELG, Splunk, AppDynamics, NewRelic, Dynatrace

Infrastructure Monitoring: Nagios, Zabbix, SolarWinds, NewRelic, Icinga, Observium, OpsGenie

Production deployment

We have discussed DevOps would help in frequent releases to production, but the major problem is how do we solve the production down time. Traditionally during releases production would have downtime. The key parameters in choosing the deployment strategy is to have a quick rollback in case of failures, near 0 down time for business users.

Canary deployment

I first read an article about canary deployment in Facebook in the year 2010. This is widely used deployment strategy for the social networking sites. Netflix uses this deployment extensively and its Spinnaker Continuous delivery tool has extensive support for Canary deployment.

In canary deployment releases are made to a subset of users/customers. For example, the new production build (version B) gets deployed in 5% of the containers. Requests from business users who are early adopters go to this set of instances. Monitor errors in production and the error count is benchmarked with 5% of existing containers (version A). If there are any major deviations observed in error counts or performance parameters, then the build is rolled back from Canary. If there are no major issues, then the remaining 95% containers (version A) are gradually replaced with new build (version B).

In my experience we adopted this in couple of the products and it worked well. The time we keep the build in Canary varies and it purely depends on the business and technical needs of the team. This option is relatively cheaper since we don’t over provision the containers. The drawback is roll-out of features is slow and at any given time there are multiple versions of the applications running in production.

Blue — Green deployment

In this deployment strategy, version B (green) and version A (blue) are maintained parallel with the same number of instances. After testing that the new version meets all the requirements the traffic is switched from version A to version B at the load balancer level.

Rollout and rollbacks are instant. At any given instance there is only one version of application in production. However, it is expensive since it requires double the resources.

A/B testing

A/B testing deployments consists of routing a subset of users to a new functionality under specific conditions. It is usually a technique for making business decisions based on statistics, rather than a deployment strategy. However, it is related and can be implemented by adding extra functionality to a canary deployment. This technique is widely used to test conversion of a given feature and only roll-out the version that converts the most.

Here is a list of conditions that can be used to distribute traffic amongst the versions:

By browser cookie
Query parameters
Geolocalisation
Technology support: browser version, screen size, operating system, etc.
Language

Conclusion

DevOps provides organizations with a set of practices based on lean and agile methods. With these methods, organizations reduce the risk developing software that does not meet requirements and increases the effectiveness and efficiency of software development and deployment. Majority of the enterprises are at some stage of DevOps adoption. Organizations should concentrate on changing the culture of the teams. The fear of failure in the team should be addressed by management. Incentivize teams who take risk and adopt early in this journey.

References:

https://martinfowler.com/bliki/ContinuousDelivery.html https://puppet.com/blog/continuous-delivery-vs-continuous-deployment-what-s-diff https://dzone.com/articles/devops-microservices-world

Please follow and like us: