Enabling Continuous Delivery: Through Process or Patterns?
Continuous delivery enables businesses to out-iterate their competitors and deliver software significantly more efficiently. Deploying to live early and often is the fundamental practice that leads to these benefits.
More and more, companies nowadays are moving to continuous delivery because they see the massive successes other companies are achieving. Enabling continuous delivery, however, is not a linear path to riches. Due to the challenges involved, there are a number of approaches available.
Keep the suits happy — deliver software frequently
One approach to enabling continuous delivery is modifying your software delivery process to avoid the impact of dependencies. The other major approach is to employ feature toggles so that unfinished features are not visible to end-users until they are ready.
Let’s start by understanding these hurdles to continuous delivery so that the trade-offs inherent to each approach can be compared.
Hurdles to Continuous Delivery
One of the biggest hurdles to enabling continuous delivery is the amount of work in progress, and the dependencies between that work. For example, if 6 developers are all working on a separate feature, how can you deploy the feature that is finished first without deploying all of the unfinished ones?
You could have all developers work on a branch which is committed to master after it passes QA. This approach, feature branching, is highly-susceptible to well known problems
Firstly, feature branching loses the benefits of continuous integration — fast feedback, risk reduction, etc. Even more harmful, though, feature branching can be a nightmare when work fails QA.
If work has been merged into master, and it fails QA, what should happen? Unmerge it and hope that no-one else had merged it into their branch? Fix the problem and block everyone else committing to master? What if two items are queued up — one passes QA and the other fails?
The likelihood of this problem grows the longer the lead time between starting a piece of work and finishing it. As an example, when you have a code review stage the window of opportunity for merging more work into master that would need to be rolled back for a QA failure increases.
Now it’s time to take a look at your options for dealing with these problems.
Modifying Your Process with Help from Kanban
If having a lot of work in progress (WIP) makes it difficult, or even impossible, to deploy frequently, then consider not doing it. Instead of focusing on getting a big batch of work complete, focusing on optimising the lead time of starting a single piece of work and deploying it.
This sounds like narrow-minded thinking — optimising the time it takes to complete one feature at the expense of many others. When you optimise for lead time, though, you’ll be amazed at what happens. Especially when you can deploy 20 times per-day.
To focus on lead time, you can start by just adding a WIP limit to each stage of your delivery pipeline; 2 features in development, 2 features in QA etc. If you do this, not only will you be able to deliver software frequently due to not having lots of conflicting WIP, but you will improve at delivering software.
Visualise WIP limits to avoid conflicts so that you can deploy more frequently
Limiting WIP is great because it highlights bottlenecks in your system. If your team are moaning that they are blocked by a WIP limit, this is actually showing inefficiency in your process. Perhaps QA aren’t QAing fast enough, so developers cannot push their work onto UAT. If so, it shows you need to hire more testers or improve your QA process.
If you are doing continuous delivery and work fails QA, you need to swarm on the failed ticket and fix it asap otherwise the system gets blocked. Can you see how this is an example of focusing on lead time?
If you complain that your team are swarming too much — fire fighting basically — this is also a sign of inefficiency in your process. It’s telling you that so much work is failing QA you are wasting a lot of time and money so you should fix it asap (I’ll explain how you can beat this problem later in the post).
Hopefully you can see that by limiting WIP you increase your chances of safely deploying more frequently and evolutionarily becoming more efficient at delivering software. But at the cost of big changes — you’ll need to accept that 6 people working on the same code base is probably not going to work.
Feature Toggles
If you don’t want to modify your process, and you want to have many developers working on the same code base at the same time, and you want to deploy software many times a day, maybe you should consider using the feature toggle pattern.
With this pattern, you wrap each feature with a toggle. In your code that might be an if statement like the following
if (featureEnabled()) {
showSectionOfPage()
}Since each feature has a toggle, you can go back to working from master and using continuous integration again, if you want to. But the real benefit is the ability to deploy more frequently.
Because each feature has a toggle, you can deploy unfinished features to live and have them toggled off until they are finished. So, when one feature is complete, you deploy all of the features, but only toggle on the one that is complete.
Feature toggle sounds compelling because you can keep working the way you do now — in theory not much changes. But the cost is adding a feature toggle to every piece of work. It has the potential to reduce your efficiency at delivering software (because it’s extra work), and it has the potential to make your code a messy feature-flag spaghetti.
Be wary of feature-flag spaghetti, but decide for yourself if it becomes too much of a problem
Choosing an Approach
I’ve blogged before about how too much WIP turns projects into a disaster, so my approach is to favour a process that supports continuous delivery and continuous improvement. I’ve seen it work well for a variety of teams and I think you should seriously consider it.
Honestly, I’m not certain that feature toggles will work exclusively as a strategy for enabling continuous delivery, because I’ve never used it that way. I think the overhead of too many feature toggles could outweigh the advantages. But I’m happy to be proven wrong and I’m not discouraging you from trying.
What I do believe is that a combination is often a good approach. Start by limiting WIP, but for big features than cannot easily be broken into small pieces and deployed stand-alone, wrap them with a feature toggle.
If you decide to modify your process you will need to seriously consider other big changes to make it work. As mentioned, work failing QA is a big problem when you have WIP limits. Therefore you will need to produce better quality software that rarely fails QA — pair programming, TDD, tighter integration with stakeholders may all be essential for you to achieve this.
Consider pair programming as a way to improve lead-time — no more mandatory code review, and fewer QA failures
If you aren’t prepared to make big changes, then you may just have to try and make feature toggles work or try to work with longer feedback cycles — maybe try 1 week sprints instead of two. You could also try hybrid approaches that balance the amount of WIP with diligent use of feature toggles.
Good luck with whichever approach you to take. You will be making significant business-level improvements.
