Value Stream Mapping for Machine Learning Driven Applications

And why it need a different mindset than building traditional software?

Photo by Austin Chan on Unsplash

In the last decade or so, every business became a technology business and every leader became a technology leader. STARBUCKS & BANK of AMERICA are today as much a tech org as META. Recent events like pandemic has only accelerated the pace at which these companies have to push frontiers and adopt technology and its practices to survive, sustain and grow.

And then came all things DATA, the next gold rush for these businesses with the limitless possibility of building AI/ML capabilities, through novel data products and ML powered applications and products.

Companies building these ML products are applying the same principles of traditional software development and thus run the risk of not producing the desired outcome in an efficient and agile manner.

We will tackle this challenge head-on by bringing out some subtle and some obvious differences while building a ML driven application, to a traditional software, by mapping out the value stream across each and surfacing the additional complexities that come when we build a ML product.

Fundamental Difference between Traditional Software and ML based Applications

Before we talk about value streams, let’s conceptually align on the difference between a ML and non ML software. This difference can be explained by the below figure.

While traditional software (Non-ML) uses known patterns(think as functions) to generate output, a ML system main goal is to learn these patterns when they are not known easily.

So, what is a Value Stream and why we need to Map it?

Value Stream is a concept taken from Lean Manufacturing theory and adopted for software development, which focuses on making flow of work visual and easy to observe.

Mapping this value stream helps in analyzing, designing and managing the entire flow (sequence of activities) from the time a new use case or a feature is identified to the time it is delivered as a value to the customer or user.

By mapping the entire value stream, one can identify point of constraints, ensure a smooth flow of work and constantly optimize the sub parts to met the global goals.

A Study of Value Stream Mapping for Traditional vs ML Software:

In a traditional software development, the value stream follows the typical SDLC(software development life cycle). The objective is to deliver the value, which moves from left to right, as quickly as possible to the customer with high quality. The feedback or information flow happens the other way around as you see in the figure below.

Value Stream Mapping for a non ML application development

The complexity increases substantially with a ML driven application development. This complexity comes from additional steps and systems that need to be created within the flow to effectively build and use a ML component in the product development. The boxes highlighted in color below are the additional parts that make up the value stream in a ML product development.

Value Stream Mapping for a ML application development

Key Learnings from Value Stream Mapping Exercise:

So, we have mapped the value stream here, now what? Here are the learnings that we can take away that will help us think differently when we are building a ML based feature or use case for our products.

1. ML products have higher lead time — Lead time is defined as the time since the request is made to when its delivered. In the ML context, the clock starts when the use case is defined. If you compare the two value stream it’s quite obvious that ML based products have much larger lead time.
2. ML use cases are not guaranteed — ML development phase is experimental in nature and can never be taken as a certainty, though the chances are high that a skilled team would crack it. It can fail during the development phase if a reasonable model can not be arrived at. It can fail later if it does not meet user acceptance criteria though the model in the early stage would have shown decent performance. It can also fail post deployment when data is changed/refreshed. That’s why ML driven products needs to spend much longer time in development and testing compared to traditional software’s.
3. ML products needs more effective cross functional collaboration — The overall flow of a ML development goes through different functional teams which creates additional complexity. Over the product, design and engineering teams ML development are high stakes activity for data engineering and data scientist /ML engineering team. This cross team collaboration is also needed post production release and hence ML products needs to be built and operated with tight guardrails.
4. ML products demands different testing focus and strategy — Testing for a traditional software are predominantly focused on code integration and testing the whole system through unit and acceptance tests. While in ML products, code is the least of worry with the focus being more on data and model testing. If you do 100% code coverage and 10% data coverage, your ML product have high chances of failing any time.
5. ML products need more Ops practices — Compared to a traditional software which needs only DevOps practices for continuous testing , integration and deployment. ML product also needs DataOps and MLOps practices to streamline the lifecycle of machine learning models in production by orchestrating and managing the movement of the data, model and systems.

Overall, building a ML driven product requires technology leaders and other stakeholders to come out of the traditional software development mindset and appreciate the unique challenges it poses.

I hope the learnings from this article will help you navigate the challenges with ML applications a little bit better next time.