Is your data science team ready for an emergency?

The story of how to quickly check your team’s practices while having a lot of fun.

How do firefighters assess their readiness for an emergency situation?
How do they ensure that everyone knows what to do in case of a fire?

The best way to check this is to organise maneuvers similar to a real life situation.

Firefighters train regularly to check and improve their procedures. They are inducting new members, they are testing new equipment, and they are building trust to each other.
Nothing can teach you more than real practice.

Like firefighters, data scientists need to perform as a team. This means introducing ways of working to new joiners, making use of the best tools and techniques they have at their disposal, and knowing each other’s skills and personalities.

In this article we would like to describe how we organised an internal Kaggle-like competition to test and assess our data science procedures.

Our small data science competition

Recently, we organised a machine learning competition at Ocado Technology to check how well equipped our data science teams were to solve real problems under pressure. We invited data scientists from our five offices (i.e. Kraków, Wrocław, Sofia, Barcelona, and Hatfield) to our headquarters in the UK. We ordered some pizzas and started a hackday.

We formed into teams and adopted only one rule: if two data scientists are working on the same team in real life, they cannot work with each other during the hackday. We wanted to encourage people to get to know each other.

We decided to use a Kaggle style competition. For people who are not familiar with Kaggle, it’s a competition where business problems, data, and evaluation metrics are defined by the organisers. Participants then have to build ‘only’ the corresponding machine learning models.

Our problem

The goal of the competition was to predict the total time-at-door for Ocado delivery vans.

We wanted to know how much time it would take to deliver groceries to a particular address. Ocado is using these delivery times to plan the van routes with more certainty and thereby open up more one-hour time slots for customers to choose from.

Similar to Kaggle, we prepared some baselines and a leaderboard where we showed the best solutions; this gave participants additional motivation to build something better than everyone else. At the end of the competition, the teams presented their findings and models. We learned a lot about our data, our practices, and ourselves.

It was a great day so we wrapped things up with the customary pint in the pub.

Five lessons learned after the Kaggle competition at Ocado

You can easily apply these lessons in your data science team or data department:

1. Hackdays are a great chance to socialise

There’s nothing like a competition to get people from different offices working together and therefore getting to know each other. People can learn about their strengths and weaknesses. We found problem-solving to be a great team-building exercise. After the event, we created a survey which confirmed that people indeed had a lot of fun.

2. Machine learning models are only the tip of the iceberg

As an organiser, you need to choose the problem wisely: it cannot be too difficult to solve in one day but still should be challenging. You have to define the evaluation metrics, gather data, split it into training and test sets, write down the rules etc. During his presentation at NIPS 2016, Ben Hamner (CTO of Kaggle) confirmed that his employees invest hundreds of hours in properly setting up the competitions behind the scenes. In all data science projects, only 5–10% of the time is spent on modeling.

3. Data science is all about iterations

During the competition, some teams over-complicated their models: they tried to check too many things at the same time and overestimated what is feasible to do during one day. At the end of the day, only working models really matter (all teams had plenty of ideas on what they would have liked to check but ran out of time).

It works pretty similarly in real life. We’ve written about this here as well. Practice and contests like this can show your team the benefit of iterative work.

4. Domain knowledge can make all the difference

Rather than trying a more complicated model, it’s better to first invest energy into understanding the metrics, analysing data, checking the distribution and outliers. The team that won the competition used their knowledge about Ocado’s business to improve their model. In real life, very often domain knowledge is essential.

5. Improve your engineering practices

Python and R are two of the most popular programming languages for data scientists. To work effectively, you need to know your tools very well, including programming languages and frameworks. If you want to rapidly check hypotheses or add new variables, you cannot be blocked by technologies.
This hackday showed us that we need to work harder on unifying our technology stack and adjusting the induction process to ensure that everyone can easily get the data, make the analysis or model and share their results with the rest of the team.

As we’ve seen, a one day hackday event can provide a very useful health check for your team. You can check how people organise their work, what tools they are using and how they are working to solve problems. But hackdays can be beneficial not only for data science or engineering teams; management teams can use them to decide training budgets, investment in tools and technology, or for forming new teams. We therefore strongly encourage you to involve your managers or team leaders in these events as much as possible.

Try to conduct similar competitions in your company. We assure you that you will learn much more than you’re expecting while having a lot of fun.

Authors: Lukas Innig, Marcin Druzkowski