Craft Team Goals — DevOps Part II
Part I is here
Last year, my team got together to discuss how we can be a more efficient team by reviewing the gap between what we have been doing and DevOps cultures and practice. While we start practicing the new ways of what we proposed to change, we had a another meeting this week to discuss how we could implement gradual changes which is one of DevOps culture.
When people get asked how they do Continuous Integration and Continuous Delivery, the use of tools like TeamCity/Bamboo/Jenkins/Octopus Deploy/Azure DevOps are usually wrong answers.
You cannot automagically achieve CI and CD by simply using tools. You may also have heard practice, like trunk-based development and merging changes at least once a day, etc, blindly following those practice does not improve team efficiency. We need to understand what problems are solved by doing them.
The purpose of CI/CD or implementing gradual changes is, for a set of changes, getting fast feedback on the quality — how good are the changes (see what we are experimenting to get fast feedback on the quality of changes by breaking silos). Once we get positive feedback, it is real progress.
In this post, I will describe a few tips and lessons I’ve learnt while I have been practicing CI/CD for the last a few years.
Reviewing Release Pipeline
In order to achieve CI/CD goals, reviewing and improving release pipeline, steps required to deploy changes, is a starting point.
Release pipeline is a set of quality assurance steps.
Every team has their own unique release pipeline, and it typically consists of code review, executing tests, building artifacts, non-production deployment, performing remaining tests, release management, production deployment and production verification. Team needs to review each step and see if there are opportunities to do it either differently or efficiently. The common practices that usually help include
- use of development tools, i.e. linters, code formatting, git hooks, etc.
- automating repetitive tasks
- splitting tasks and running them in parallel
- build as code, configuration as code, infrastructure as code (and version controlled)
- use of mocking framework
- making execution of tests as cheap as possible, i.e. fast and being able to run locally, use of headless browsers, etc.
- extensive and separated tests suites
- what we are experimenting
- modular design
It is also worthwhile to benchmark the performance of your release pipeline before making any improvement so that you can see the benefits later. The metrics are unique to your release pipeline and there are a few examples for your reference,
- How often do developers commit changes to a shared branch
- How long does it take to create release package from code commits
- How long does it take to deploy release package to an environment
- How long does it take to finish tests
- Code test coverage
- What is our lead time for release management and service delivery to approve production deployment
Development Practice
No matter how fast the release pipeline is, large-batch changes prevent you from achieving CI/CD goals and make releases complex because they
- increase risk of errors and the need to rollback
- make it difficult to figure out what changes cause issues
- make code integration/merge painful
Avoid large-batch changes when possible. Small changes that are high in cohesion makes it easy for peers to review and help easily solve merge conflicts as early as possible (fast feedback again!). In practice, there are tips on how safely (no breaking changes) and comfortably make changes,
- incremental changes where changes are not made to the existing interfaces but new interfaces are added and used to replace the existing ones when done (open for extension, and closed to breaking changes)
- refactoring or/and adding unit tests prior to making any changes for the new requirement, and, if possible, release the refactoring separately as early as possible so that the changes are not breaking existing behaviours
- use of feature toggles
- dark launching, i.e. new but unreferenced code
- decoupling changes to minimise dependencies, so that less coordination is required when deploying changes, i.e. dark launching, interface or contract based development
- infrastructure or application capabilities that support tests of changes in production environment, i.e. staging slot, ability to load staging scripts, etc.
If we follow those practices, the branch that contains changes is short-lived and changes can be merged at least once a day and potentially get deployed.
A branch is, by-design, intended to hide change in one part of the code from other developers. It is anti to CI
As ever, the definitive point is the testing that happens at the point of merge to Trunk. It is only at this point that you can honestly say “Yes, my change works with everyone else’s.”.
This often means that we are happy to deploy changes into production that are not yet complete, but don’t break anything!
