Dual-Track Agile Introduction: Why Messy Leads to Innovation

This is an introduction to using experiments in dual-track agile.

Dual-Track Agile is an IT development methodology where figuring out what to build is as important as the building process. You start with a discovery track to find out if a product idea is good and if it makes sense to build. Successful findings from the discovery track are added to the backlog of the delivery track.

So how do you know if your product team has a good idea? Luckily, there are a variety of methods that you can use to figure it out. Personally, I am a fan of the experiments, because you evaluate ideas based on behavior before you invest in them.

With Dual-Track Agile, the development team participates in the discovery track — often led by the lead engineer, the UX designer, and the product manager. In other IT development methodologies, the IT teams are only a part of the delivery track, so they don’t get the valuable knowledge from talking with users and performing experiments.

The Dual-Track Agile process is demonstrated in the gif above.

Now Things get Messy

While Dual-Track Agile is my recommended way to develop products, it’s not very clean. Think about classic project management, and how it’s anything but easy. Now step it up a notch with an extra work track for performing research. Dual-Track Agile raises complexity in many areas, especially when it comes to planning and resource allocation. Since most product managers are already familiar with delivering production-ready code in the delivery track, I’ll keep my focus on the discovery track.

How to run a Discovery Track

The purpose of the discovery track is to work out if a product idea makes sense to build. During the discovery process, the product team should collect answers to the following four questions:

• Will users use it? (Value)
• Can users use it? (Usability)
• Can we build it? (Effort)
• Can we get internal support? (Stakeholders)

Let’s take an example from product development at Trustpilot, where I work as a Product Manager. Trustpilot is a review platform that helps companies collect reviews, and consumers shop smarter. Reviews are displayed publicly, so consumers can read about companies before they buy products or services from them.

Objective and Brainstorming

At Trustpilot, we get an overall objective for each quarter. The objective is outcome based (e.g. increase leads) rather than delivery based (e.g. build a specific feature). Outcome-based objectives are fundamental to Dual-Track Agile product development.

Our objective in this example was to increase retention.

The team, comprised of developers, designers, and myself, came up with a variety of ideas that could help us reach our objective. One of the ideas we came up with was the digest email.

The idea was to send the digest email to users two weeks after they had written a review on Trustpilot, to provide them with an overview of how many people have read and liked their review. Digest emails work well for other companies, but we had to work out if a digest email would make sense for Trustpilot. Would it give users significant value? We used the process of discovery to reach an answer.

Framing an Idea

Before discovery, we had to frame our new ideas to ensure that we understood the basics of our product concept. In the framing process, we rejected some ideas because we could not argue that they were important enough, but eventually, we came up with this frame:

FRAME: Digest email

• Why are you building this?: To get users back to Trustpilot’s website after writing a review
• Who is it for [minimal customer group]?: All users, but especially frequent users.
• How would they user it?: Users will receive an email, so they know how many they helped by writing their review. This praise will motivate them to return and even use the platform more.

There are more detailexd ways to frame your ideas, like the opportunity canvas, but we chose to go with the three questions above, as they are the core in all idea frameworks.

Once we finished setting an objective, brainstorming, and framing the idea, we were ready to move on to discovery.

1) Value: Will Users use it?

The first question is if users will use or buy your idea. It is rarely a question of whether an idea creates value,the question is how much value the idea creates. In some cases, the question could be how much customer are willing to pay for what you are developing. At Trustpilot, our question was how many users would return to the Trustpilot review site after receiving our digest email? And the easiest way to test this out without building anything? We faked it.

To fake it, we randomly selected 5,000 Trustpilot users.

We created a mockup of a digest email, shown on the left.

The challenge with the mockup was that we didn’t have accurate statistics on who had read and liked reviews of specific users.

We could have made up numbers, but that wouldn’t have been ethical. (Ethics often get messy with Dual-Track Agile, but that’s another topic.)

However, we did have the tracking statistic for how many people loaded a page on which the review might be shown. So we were able to estimate numbers. To have a clear conscience, we even put our uncertainty in the footer of the experiment email:

At Trustpilot, our principle when it comes to running “will they use it” discovery is that we validate our ideas in the fastest, cheapest way possible. So to keep costs down and to move efficiently, we removed the more complicated part of the email which displayed reviews from other users with similar opinions. That resulted in this simplified version:

Results:

After sending out 3,000 emails with estimated statistics, we got the following results:

• Email opens: 52%
• Retention: 21%
• Re-activated: 5%

We found these results very promising, and our verdict was that our idea also provided users with significant value.

2) Usability: Can Users use it?

It is extremely important to talk to the end users who will be using your product.

The usability question in discovery is typically whether a user can understand and use a complex product or feature. The trick is getting the design right. This can give great insights into whether a product or feature will work. In Trustpilot’s case, the idea was simple, so we wanted to know if users understood the email and how it made them feel. In user interviews, we presented the following mockup:

Being presented with this, it seemed as though all users understood the email, and we had users saying things like:

• “I hate to admit it, but this kinda makes me feel important.”
• “It makes me feel popular.”

One advantage of interviews is that you learn more than you ask for. And we definitely did. Turns out, the placement of the “found useful” number below the number of people who read the review was quite problematic. Users were disappointed that of the 36 people who had read their review, only five had found it useful. The “found useful” took value away from the “reads”.

Results:

All users could use the digest email, and they thought it was easy to understand. The email content made them feel important and popular, which confirmed the value of the idea. We decided to remove the “found useful” numbers from the email to avoid deflating the value.

3) Effort: Can we Build it?

This is more of an engineering task than a product management task. Though it is seldom a question of whether product teams can build something, but rather how much effort it takes to build both in relation to man hours and finances.

We knew that our digest email was not the simplest of tasks and it would take a couple of weeks to build. But based on our prior findings, we concluded that it was a feasible investment.

4) Stakeholders: Can we get Internal Support?

I don’t like the words “stakeholder management” because it sounds like you have to persuade people when really it’s about getting input from others who have more information than you. For example, legal, sales, marketing, and support departments, all have valuable knowledge that you need to take into account. These stakeholders may present information that you didn’t know about, and in the worst case, this information might have to stop the experiment altogether.

In our case, we presented the digest email idea to our sales and legal departments, as well as top management. Honestly, we had expected everyone to just say ‘yes’. But it turned out that there were two concerns, namely a risk with our enterprise customers who might be reluctant to share data. Secondly, we couldn’t send the email to users in all countries because of different email/marketing laws.

Results:

We decided to push forward with our idea, but we added an opt-in to the digest email being sent to users in four countries with strict marketing laws. And to avoid scaring enterprise customers, we decided that we’d roll out the product gently so we could fix issues fast, if any would emerge.

Moving to Delivery

After having positive results from all four questions in the discovery track, we decided to give the digest email a ‘go’, and moved our idea from the discovery track into the delivery track.

There were several weeks of building, followed by usability testing, and in the end, we had a real digest email geared for Trustpilot users.

We saw positive results immediately. After receiving the digest email, more users started going back to Trustpilot’s website each week:

We even got great feedback from satisfied users on social media:

Why is Discovery so Messy?

In theory, adding a discovery track to an agile process is clean and useful. But the strength of Dual-Track, namely research and avoiding expensive failures, can also be the weakness of the method. You see, pure delivery is like a factory because you simply have to estimate how long it takes to produce code. But, when you add a discovery track, you make staffing and planning more difficult and complex. That might make some people in your organisation raise their eyebrows.

Making a Discovery Plan

You might be able to ease the minds of managers or other colleagues by showing them plans for your feature or product ideas. But if they expect to see traditional road maps or timelines, then you’ll have to disappoint them. With discovery, you won’t know what experiments will be a success and your focus has to be on the outcome. For an organisation used to measuring written code and value points, this may create an uneasy feeling of unpredictability.

And discovery can create even more entropy. Sometimes learnings from a discovery experiment can require new experiments, in which case we need to extend the current discovery process and run more tests and interviews in order to verify the product idea.

This messiness is an inherent part of Dual-Track Agile — because you plan with objectives as opposed to features. Sometimes upper management may insist on a roadmap because that’s what they’re used to. In that case, I have a few suggestions.

One option is to map out the experiments that you’d like to perform in order to achieve your objective. That doesn’t mean creating a timeline, as that is quite tricky to estimate. It depends on traffic/volume on the test page, how close results are before reaching statistical significance, and the complexity of the test. Depending on which part of the product you’re experimenting with, it can take anywhere from an hour to several months.

Another option is to create a long term plan of objectives/themes in collaboration with upper management, so they have some input and control of the process.

Organising Staff

All team members working together on the same delivery task

Staffing is perhaps the messiest part of Dual-Track Agile. Having multiple tracks creates complexity because team members have to switch between different mindsets faster.

Same team members work ahead with discovery

My team at Trustpilot appreciates when we can work on the same task together, as it makes them feel interdependent, it’s motivating, and it’s efficient.The moment you start working on a discovery idea, you split the team up a bit.

Small discovery tasks can split the team even more

And since discovery tasks are often small, teams are usually divided into smaller groups or tasks. This can be challenging to manage.

The discovery process can become more chaotic if many ideas fail. Failing is of course the strength of Dual-Track Agile because failed experiments are much cheaper than releasing features with no value. This ‘fail fast’ process is also educational, as you learn how to build better products.

But sometimes you might run out of delivery tasks in the backlog, and the whole team can work on discovery together. This could be fun if everyone gets to collaborate on the same form of discovery. Then again, keep in mind how you use the precious time of developers.

In Trustpilot’s case, most of our discovery is made in the front-end because we experiment a lot with user experience. That means we don’t really have discovery jobs for our back-end developers. Instead, they take advantage of the time, by e.g. repaying of tech debt. It creates value, but it divides up the team too. Overall, Dual-Track Agile makes staffing challenging and requires pro-active developers.

Why I Always go With Messy

Despite the increased complexity and chaos, I’d still choose Dual-Track Agile any day over other traditional methods. Here’s why:

• You only build a product that you know is valuable for real objectives, and therefore have significant business impact.
• You never start to build something that does not turn out to be valuable. This includes HiPPO ideas, which often get built but then fail with no one to blame.
• Discussions last forever if we have no data. We can kill ideas pretty fast. A discovery experiment kills or thrills an idea instead of having endless meetings.
• The development team is involved in business decisions. This way, they understand why you’re building a new product, so there are fewer misunderstandings and in turn, fewer mistakes. It’s motivating for development teams to have an influence on product decisions. Motivated staff are more efficient.
• You learn more than you expect. So instead of just being a factory, the team gets better at building products, and engineers are really clever people who then end up coming up with the best ideas.