How to effectively manage Complex Code Bases

Source Control is an integral part of software development. Our life gets really easy when the code is properly stored and managed, specially in scenarios of continuous releases by large product teams.

Microsoft’s product teams which work on development and testing of Windows alone consists of more than 2000 employees. With such large teams imagine what nightmare it would become when code is not maintained and releases properly.

I work in a team of almost 25 developers across geographies. While working on a product being deployed as a service on cloud, CI / CD with proper source control is cornerstone for our team. Recently we started introspecting our ways of managing code releases more seamlessly having proper versioning and a quicker development to production turn-around time for our code.

The upcoming scenario of release management and its process requirements would be very similar for almost every software development company.

Let me start by stating the obvious. Our source control software is GIT.

Usually Code Management is done in 4 different types of branches on GIT

• Master / Develop branch (Sometimes both)
• Feature branch
• Release branch
• Hot fix branch

Our core product is Teradata Database Engine + Machine Learning Engine as an offering on Azure.

Let’s consider a scenario where we are working on integration of Viewpoint and Unity (Teradata’s proprietary Add On products) with the Database engine.

Note- Above is just a hypothetical situation. Database with all the supported add-ons as an offering on Azure is already made generally available.

These integrations would generally be 2 features that we would work on, in parallel.

Also let us consider that integration of Viewpoint would be made available in production a month after the integration of Unity, as Viewpoint is a complex beast to tame (again, this is hypothetical).

Our overall requirements from the Release Process are as follows.

• Our approach should be simple enough for team members to quickly get onboard with and start with their code contributions.
• We should have the flexibility to make quick fixes on to the released code without having incomplete and unnecessary code getting deployed to Production.
• We would want to track our releases properly and would want to support multiple releases at the same point of time as not all of our customers would be ready to hop on with the latest release.

Let us start with Git Flow approach on how we would manage these releases, post which we will touch base the Microsoft’s release flow strategy. Finally we would talk about what we are planning to incorporate for our team.

GIT Flow Approach

This approach was coined by Vincent Driessen in 2010 and is still quite popular with a majority of Product companies. Let’s frame our above scenario in the Git Flow approach.

Master and Develop Branch

• Our main stable production code for enabling Teradata offerings as a service on Azure would be present on Master branch. It stores the official release history of all the features released till date.
• We would have a Develop branch which serves as an integration branch for merging releases into master. Develop branch is forked from the master branch soon after the creation of master branch.

Feature Branch

• We will develop our code for Viewpoint and Unity integrations as 2 different Feature branches and merge it into develop branch.
• Multiple developers can work on their own sub-feature branches based on their granular task and merge code into the main feature branch to complete the feature.
• Once a feature is Dev complete, we make it up to date with Develop to have any new checked in feature merged with it and then get a sign off from QA. After successful QA, the feature branch gets merged back into develop.

Release Branch

• As in our hypothetical case, we are releasing Unity integration first, we fork a Release branch from develop after the Feature Branch is merged in to develop.
• This Release branch becomes our production candidate and gets merged to Master as per release schedule with a release label.
• Development of other feature is not affected as they get merged only into develop until their release is due.
• It becomes easy to look at release level changelog that went live using this release branch approach.

Hotfix Branch

• Consider a scenario where deployed Teradata Database engine is not able to communicate with Unity in a particular Azure region (assume the China region). This is a bug on a live offering
• To tackle this, we would create a Hot Fix branch from Master and apply the fix to this issue on the Hot Fix branch. After testing, we would merge the fix to both Develop and Master Branch.
• Merge on develop branch is required to avoid overwriting the fix when a new feature is merged in master through a different release branch.

How does Git Flow approach help us??

• We get a good clarity and control on what features are made live and when. We get proper release history management in our code base for different features.
• Development of our non-release candidate code does not get affected due to an impending release as that code is always checked in to Develop.
• We can enforce greater check-in sanctity by making individual Feature branches as protected Git Branches and merge code from a developer specific sub-feature branch into the main feature branch via a pull request.

How does the Git Flow approach increase our overhead??

• For this approach to work flawlessly, we need to be very vigilant that all the hot fixes are merged to both Master and Develop branch. If we miss out on merge to develop this will cause the issue to resurface in production in later releases. Small human error can cause a major production impact.
• With so many branches in play, it would get cumbersome to release small feature upgrades.

Microsoft’s Release flow

To reduce the complexity of Git Flow and increase the velocity of code moving to production, Microsoft as a whole is moving towards a “One engineering system” with internal teams following a simpler approach to branching than Git Flow. One of the teams is the VSTS team.

While git flow approach is pretty solid in managing code for releases, for large teams it becomes a bit cumbersome to manage so many branches and hierarchies. Also using Git Flow leaves us with the chances of having frequent merge conflict due to the way we merge hot fixes (more on it in the disadvantages section)

Let us see how our scenario of integrating Unity and Viewpoint with database engine fits in Microsoft’s Release flow. Here we typically have fewer types of branches.

Master Branch

• This branch is the central backbone of every merge. It’s the source of truth in terms of code of Teradata as an offering on Azure at that particular point in time.
• Individual topic (like code for integration of viewpoint) and hot fix branches are merged into master branch and releases are branched out of the master branch.

Topic Branch

• Every developer would make these branches from Master and actively work upon it. This branch would either contain a small chunk of code for a particular feature or a minor bug fix etc.
• In our case any development for both the features would be done via different topic branches worked upon by multiple developers.
• A merge to master through a pull request would be performed from a topic branch when the code it was intended for is complete.
• According to Release Flow approach, we usually would not have a child topic branch merge into a parent topic branch and then to the master. This is done to reduce complexity.

Release Branch

• We would make a Release branches out of a master branch when code till that particular point of time needs to go to production. For example when we are done with the development and testing of integration of Unity with database as per plan, we would create a new release branch.
• Usually in Agile teams, a release branch is made at the end of each sprint and tasks in a sprint are defined such that they can be made as Production deliverables at the end of the sprint.
• When we are at a juncture of having a new release, we would no longer require an older release branch. We create a new release branch from Master as it has the latest code and abandon / delete the older release branch.

Hot Fix Branch

• Hot fix branches would be generally made when we would want to patch a bug on urgent basis on production code.
• Here we have a major difference on how hot fixes are handles as compared to Git Flow.
• When we have a bug in this case, we create a normal branch from Master, follow the usual course of fix then test, and merge it back to master. We then selectively cherry pick this change to the release branch as well.
• In this way we make sure that the fixes are always present in the master branch.
• As a live fix would only concern code which has been successfully released, chances of merge conflicts are minimal.

How does Release Flow approach help us??

• Our Master branch will always up to date in terms of the latest code as all the bugs, hot-fixes and feature development code is first checked into Master branch. This gives greater control and confidence in managing releases
• As we just have a master branch here and no develop branch, it is less complex to manage specially when our team grows.
• This approach fits in with our team as we follow the Agile Methodology of development.

How does Release Flow approach increase our overhead?

• As everything comes out of and goes into Master, it would be difficult to stop changes from going to release branch which are not intended to be a part of the release.
• Release timeline formation and work segregation will have to be done flawlessly to prevent incomplete feature(even though it is disabled) moving into production through a release.

Best of Both Worlds

As a lot of other product development teams out there, we were also not quite mature in terms of having our releases very well defined. Given the degree of exploration we do to get the best out of Azure for providing our product as a service, it is gets difficult sometimes to properly timeline a release date for a feature.

Knowing our requirements and our learnings of both Git Flow and the Release Flow, we decide to take the middle way out. Extract the best of both the approaches.

We plan to follow the Microsoft’s Release Flow but with below two major changes

• We create a parent topic/ feature branch and merge code related to a feature in the parent topic/feature branch using the usual process of a pull request. Only when a feature is completely developed and tested, we merge it master.

• We create a hot fix branch from the release branch and cherry pick the changes into master branch after the fixes are merged back into the release branch. This way we make sure that only the fix goes into the release and it also into the master branch.

Following a proper approach to manage code not only helps in reduction of bugs but also helps in planning and maintaining releases properly. Using this approach we can now look forward to moving towards a monthly release cycle and CI/CD pipelines accordingly.