Running experiments stopped us getting things done

By Kevin Hall

Kevin Hall shares his presentation on how running experiments stopped us getting things done
As a product team we strive to deliver value to our travellers.
We also believe that eliminating waste and fast delivery are fundamental.

These two statements form part of our core beliefs. We want to deliver value fast. We believe experiments do have value to the user and help guide product direction. However, when we do experimentation and A/B testing what impact does this have on quality, waste and speed of delivery?

In my team, we own features on the homepage and elsewhere in the flights funnel. We strive to build solutions that help travellers. Like all teams in Skyscanner, we’ve embraced experimentation to help us understand and build products that our travellers want. It offers methods - using actual user data to see if what we build solves their user need.

While embracing experimentation, we realised that our existing Kanban flow, with its simple visualisation of well laid out planned tasks and controlled WIP (work in progress) was quickly becoming complex. Our throughput slowed right down and the board was basically a total mess.

Through sharing this experience, I hope to provide you with one potential way to get through it so you can do Kanban and run experiments in perfect harmony (we all like to dream).

The Beginning

We followed Kanban. We had a simple board, cards move from left-to-right, priority based on ‘how do I move the furthest right-hand side card to DONE’? And so, the flow flows. The team knows what’s going on. Everyone’s happy.

Then we started running experiments (namely A/B tests and multivariant tests). As the business started taking a more scientific approach … we started running more experiments … we wanted them to be visible, so we added them to the board.

Great, we added a new column called ‘Experiment’ which comes after ‘Deploy’ and before ‘DONE’. Makes sense.

And we started. Working to get as many experiments running as we could, learn as much as we could and impress everyone with our ability to ‘move fast’.

We made a start and it was not long before we had an experiment running ... great. In the meantime, we worked on building other experiments and getting those running too.

Everyone in the team knows that getting the experiments running is the priority and the cards being worked on in “In Progress” are all related to getting new experiments up and running.

The team is happy and working with a clear purpose.

The team have done great and we have 3 experiments running.

Each experiment runs for a minimum of 1 week. We see different traffic behaviour depending on the day of the week (users on Monday behave differently to those on a Saturday), so we need traffic data from at least a full 7 days to ensure we get a clear indication of what is working or not in an experiment. Some experiments run for 2, 3 or 4 full weeks.

Getting back to the Board, we had 3 experiments running which meant that we could now can start some other work. Great, the team was really shifting.

So we pulled new tasks onto the board. These included other experiment ideas as well as general BAU work and non-experiment tasks.

We now had 4 experiments running.

At this point we started to see a new behaviour emerge: in our daily catch ups we started asking ourselves about the experiments and their status.

It goes something like this:

“When’s experiment X due to finish?”
“Not sure, let me look at the card… its next Monday.”
“Right, let’s move on.”

In the meantime we continued to pick up new cards and our inventory flowed across the board. It’s not long before we had 8 experiments running.

Wow that was awesome.

For cards that are non-experiment related they just jump from ‘Deploy’ to ‘DONE’. We were getting stuff to DONE and the team was happy.

The Middle

This is where things started to change.

We now had 8 experiments running, of these, 4 were actually finished and could be analysed, but we had been so busy on the work ‘In Progress’ that we hadn’t been able to analyse them.

Ahh — another new behaviour, not so good.

In addition the only person who could do experiment analysis is the PO (me). And some of the 8 experiments were complex and would take 2–3 days to analyse.

Mmm — not good at all.

It started to dawn on us that getting experiments up and running was NOT the same as delivering value. In fact, running experiments had stopped us getting stuff DONE. What had happened to our simple organised board?

We were grumbling loudly. We found ourselves in a bad state with the following symptoms.

  • Lots of experiments but no actionable insights
  • Confusion about when experiments should be ending
  • Unclear next steps for finished experiments
  • Not enough stuff getting to DONE

So we stopped and asked ourselves ‘why’?

As a team we ran a retrospective session (using Trello) on what was causing these symptoms.

We agreed on the following 4 remedies:

  1. Unblock the experiments which are ‘ready for analysis’.
  • I ran training on analysing experiments and paired up with 4 fellow team members who wanted to learn.
  • Spending time on active learning with team members enabled the team to scale up by reducing a known constraint.

2. Make it visible when an experiment ends.

  • We added a “due date” to each of our Experiment card titles so that anyone looking at our Board could see right away whether it was still running OR ready for analysis. This immediately cut down on some wasted discussions during daily catch up.

3. Make it clear when an experiment is being ‘analysed’.

  • We added a new column (I know, but it helps to visualise our work better) for cards that were in Experiment Analysis.
  • This really helped all the team see who was doing the analysis and the status the Experiment was in.

4. Introduced a ‘Next Steps’ step.

  • The team made it clear that not knowing what happens after an experiment is analysed was a frustration.
  • To remedy this every experiment now has to have a follow-up card called ‘Next Steps’.

To expand on the ‘Next Steps’ card; it contains a recommendation and options for the experiment. We also run a short briefing for the team where we review these and finalise what needs to be done.

This could be as simple as removing the experiment code. Or creating a new Epic in Jira for another experiment as we iterate. Or it could be that the experiment was a winner and we are now working to scale it up and make it a fully productionised feature, or even running a more enhanced version of it. Each of these require work and need clarified to to the team.

Only when these individual tasks have been done can the experiment Epic be moved finally to DONE.

The Present

This brings us to the present day.

We are at a point where our board is not perfect, but the changes we made has helped the team run experiments, we know at any point what the status is and also what follow-up work is needed.

We have added some complexity to our board, but it’s all focused on enabling us to run experiments, getting stuff to DONE and knowing what the hell is going on!

The key take away

Experiments are a different type of task to traditional Kanban. They don’t follow the normal pull model. You start an experiment then leave it running and get on with other stuff. Then you need to jump back to it at a moments notice and then spawn follow up tasks that jump onto the board leap frogging existing tasks.

As a team be OK with adapting your flow to suit the way you work and the type of tasks you are working on. We realised when we added experiments to our flow that cracks (pain points) started to appear. Like any agile/lean team we acknowledged the frustrations and asked ourselves ‘why’.

Remember! Sign up for our Skyscanner Engineering newsletter to hear more about what we’re working on, interesting problems we’re trying to solve and our latest job vacancies.

About the author

Kevin, an enthusiastic runner (flat & hill) and cyclist, hails from the Scottish Borders, and is a Senior Product Manager at Skyscanner. His first foray into digital was when working as a VSO (volunteer Information Systems Advisor) in Sri Lanka for the National Science Foundation.

On returning to Scotland he joined as a Systems Administrator in 2000 just before the dot-com bubble burst. He has worked with teams (at, and now at Skyscanner) who are all focused on delivering value as quickly as possible to users.

Kevin Hall, Skyscanner

SEE the world with us

Many of our employees have had the opportunity to take advantage of our Skyscanner Employee Experience (SEE) — a self-funded, self-organized programme to work up to 30 days during a 24 month period, in some of our 10 global offices. There is also the opportunity to work for 15 days per year from their home country, if an employee is based in an office outside of the country they call home.

Like the sound of this? Look at our current Skyscanner Product Engineering job roles.

Some of the team at Skyscanner