A case for Continuous Delivery (CD)

In my previous role, I was responsible for vision, architecture, design and construction of a new data platform (read data lake). The vision (generalization) is to

• build this platform once and deploy globally to internal customers.
• enable internal customers to customize this generic platform through configuration and extensions to meed their needs.
• build the functionality that is absolutely needed for the customer (no second guessing or presumptive feature creep). Idea is to avoid cost of delay and cost of carry.

The vision is sound but very challenging. How to recruit the right talent? Where should we hire? How to structure the teams? What is the right size? Feature teams or component teams? What should be the tech stack? Batch or streaming architecture or both? Should we use Hadoop distributions? How to measure the success? What is the Definition of Done? What should be the frequency of delivery? etc. Where to start?

As you can guess, there are lot of unknowns. It was very tempting to start with the process and agile\scrum ceremonies. Some of the teams (and Organizations) I had known always start there. It is not only tempting but also the easiest way to start. However, we focused on the outcome and the fastest way to get there. After all, any feature function is a hypothesis until proven otherwise. So the faster we deploy the feature, the sooner we get the feedback. This focus on outcome and eagerness to get feedback had what prompted us to adopt Continuous Delivery. So you see, CD was a necessity for us to succeed.

While the choices I’ve made to address all of the previous questions warrant their own posts (hopefully I will get to that in future), I plan to share the following as a multi-part series:

• my journey through CI and CD
• a template (and recipes) for CI and CD
• the choices I’ve made and thought process and rationale behind those choices

NOTE: I don’t consider myself as an expert in CI or CD. I’ve been through this journey and idea is to help you jump-start your CI and CD journey by sharing mine. I also hope to learn from you all so question my choices, provide suggestions, comment and share these posts.

Read the next part in the series: Understand your Value Stream: The First Way - Part 1

Reference

The following books have been very helpful in this journey. I highly recommend you to read both of them.