How to implement dual-track Discovery & Delivery in a product team

At ING Analytics, we develop AI-driven products that make peoples’ lives easier. Our team works on Holmes, a product that makes information sharing easier within ING Wholesale Banking. With Holmes, our front office gets recommendations and can search easily through the information needed to serve customers better. Want to know more? Watch the Holmes video here.

Two years ago, our team was at a pivotal point. Our prototype had received overwhelmingly positive user feedback. As a result, we were able to secure funding to build an MVP. And to this end, we could extend our team to a full product squad of eight people (a product manager, a UX designer, two data scientists, and four software engineers).

Since the team was new, we needed to determine our way of working together. The way-of-working that resonated most with us was dual-track discovery & delivery. In this approach, the work is divided into two equally important tracks:

• Discovery: experimenting to find out what we should build
• Delivery: building production-ready software

We immediately loved this way of working because we recognized that:

• It puts equal focus on both delivery (e.g. building the MVP) and discovery (e.g. finding out how we could grow the product going forward).
• It helps you structure research work (e.g. data science exploration, UX research), which is notoriously hard to put in a scrum process.
• It allows for both experimentation with (sometimes crazy) new ideas that are quite likely to fail, as well as building robust and reliable software.

While there is quite a bit of information describing the idea of discovery & delivery, we found little practical information on how people have actually set this up, especially for smaller teams. With this article, we want to share how we implemented dual track in our team and what has worked well for us.

1. Define your Objectives and Key Results together as a team

When working in dual track, it is very important to have clear objectives and key results (OKRs). After all, these OKRs determine what you focus on in discovery, which in turn affects what you work on in delivery. For example, suppose one of our objectives is that users find new information. An idea for the discovery track could then be to research AI-based recommendations.

Because of the importance of OKRs, we involve the whole team. When you create OKRs together, you can commit to them together. This avoids lengthy discussion about prioritization later on because everyone knows what is important to achieve the desired outcomes.

2. Come up with many ideas and prioritize

Once the OKRs are clear, we have a brainstorm with the whole team to identify the assumptions with respect to our objectives. Each of these assumptions concerns either feasibility (can we build this?), desirability (do users need this?), or viability (does it work for the business?). As a start, we encourage everyone to come up with as many assumptions as possible, any idea is great at that point. After that, we prioritize the assumptions along the following two axes.

• Impact: how critical is this towards achieving the objectives
• Known/unknown: how confident are you that the assumption is correct?

The assumptions in the top right quadrant of the grid are the most important ones: they will become your discovery items. The ones in the top left quadrant can be moved to delivery: you are already confident you need to build these and that they are critical to your product. The ones in the bottom two quadrants you can deprioritize, because they are not critical.

3. Write clear experiments

For each assumption, we then define the smallest possible experiment we can do to test the assumption. We specify:

• Assumption: What is the assumption that we are going to test? Example: We can offer relevant recommendations using Named Entity Recognition (NER)
• Approach: In a few words, what is the experiment we will do to test this assumption? Example: Apply NER to 50 documents. From this, generate 10 recommendations and show them to 5 users during an interview.
• Success criteria: One or two quantifiable rules that allow you to judge whether the assumption is validated. These are often difficult to define, and hence are frequently a source of debate in our team. Such debate is great, however, as it sharpens the common understanding of what we want to achieve. Having good success criteria brings a lot of clarity and limits the risk of scope drift (i.e. tickets where you start working on one thing but end up doing something different altogether). Example: 8 out of 10 recommendations are indicated as relevant by our users
• Timebox: How long should the experiment take? Experiments should be as short as possible, so our experiments never exceed more than two days and preferably take less than a day. If you cannot do an experiment within this timeframe, try to split it up into smaller experiments. Like success criteria, time boxes are often a source of debate, as they really force you to think about what you can do in an experiment that is as small as possible. This helps sharpen the common understanding of the experiment. Example: 16 hours

The experiments are refined by the experiment owner (which can be anyone in the team), and we discuss them in our product square. This group consists of the tech lead, the data science lead, the design lead and the product manager (because data science is such an important element of our product we have formed a product square instead of the more commonly known product triangle). By discussing the experiments, we ensure that they get feedback from every discipline. Once the experiment is clear and agreed on, we move it to our discovery backlog. From there, anyone in the team can work on it.

We organize the discovery backlog using kanban. This helps us with the high degree of uncertainty in discovery. After all, the outcome of a ticket often affects the tickets that follow it. Using kanban allows us to continuously incorporate these outcomes and pivot when needed.

Every member of the team works on discovery items and everyone is involved in brainstorming for ideas. In general, we spend 50% of our time on delivery tasks and 50% of our time on discovery tasks. Of course, deadlines may occasionally require a different time distribution, but in general, we consider discovery as important as delivery.

4. Consciously decide to build, kill, or do more research

After finishing your experiment, or upon exceeding the time box, there are three possible outcomes.

• Success: move to delivery
• Failed: stop working on this idea
• More research needed: create and refine a new experiment, prioritize it in the discovery meeting

To determine which outcome would be the best fit, we carefully consider our results and how they relate to the success criteria. We advise to write these considerations down briefly and archive them, as it:

• Forces you to be very specific in how the results relate to the success criteria;
• Helps to archive the results, allowing you to easily go back to earlier experiments; and
• Facilitates sharing the learnings with others

In addition, we also regularly present the results to the rest of our team (usually during a demo), to make sure everyone is up-to-speed on the learnings we gained in the discovery track.

5. Once ideas are validated in discovery, build them in delivery

Once we have validated all risky assumptions for a particular feature, we are ready to start building it on the delivery track. In this track, we organize our work using Scrum. Since scrum is widely used and well-described, we won’t go into the details here. Instead, we will discuss two points specifically related to delivery in the dual track.

First, we keep the discovery board separate from our delivery board. We prefer it this way because the two tracks require such radically different mindsets. Discovery is all about coming up with new ideas and then trying to validate them as soon as possible. Delivery is about committing to an idea and creating working software. Having these completely different worlds on the same board just did not work for us.

Second, delivery is not the end of the line. Once a feature is in production, we get to see how users interact with it. This often leads to new ideas or assumptions, that then find their way back to our discovery track.

Conclusion

In this article, we have described how we set up dual-track discovery & delivery in our team. For us, it has been working very well. Even though we spend less time on delivery (50%), we are actually more effective in what we build and we have a better view of where we want to go with our product.

We would love to hear about your experiences using dual-track, so please leave your comments or questions below.

Cowritten by Nanne Aben, Lead Data Scientist