Supporting Data Science Strategy Through Internal Hackathons

Introduction

In the last blogpost we talked about some aspects of Machine Learning culture and organization at TUI. In this blogpost we describe how we organized a competition that not only provided insightful and practical solutions to our problems but also resulted in greater usage of our technical infrastructure, wider adherence to our developmental practices, and stimulation of network between our data scientists across the globe.

The Challenge

We decided that the hackathon problem had to be challenging but solvable so that people feel that they are being called upon to exhibit the best of their craft for successfully solving a problem. Moreover, it must be relevant to their work and require a wide-ranging synthesis of their preexisting skills to produce useful and elegant solutions in the allotted time.

We chose the problem of displaying suitable hotels based on user search information and business objectives. A good solution would have to characterize the customer preferences with a reasonable degree of precision and completeness. But the diversity of the relevant factors makes such a solution very difficult to find. Our customers can be anyone from an independent traveler to large families with highly varied budgets and age profiles. Similarly, we offer a wide-selection of hotels that span the globe and cater to a variety of customer requirements. Moreover, successful accomplishments of business objectives as well as effective communication with the non-technical stakeholders is vital to the success of any data science project. So we explicitly graded the teams on these aspect as well.

The participants were asked to provide a ranked list of hotels that meet customers’ preferences and that the customer is likely to buy, while simultaneously fulfilling some clearly defined financial objectives. This solution should be competitive with a preexisting baseline solution inspired by our existing ranking methods. Participants had to pitch their work in under 15 minutes to a jury of predominantly non-technical senior managers.

The ranking was evaluated based on a variant of the nDCG score to measure its relevance to customers’ preferences and an additional score to measure business objectives. The jury voting was based on individual votes of the jury members. We used weighted mean for selecting the overall winner.

Setup

We had four teams of four or five participants each. The teams were organized so that their members came from different domains within the company, residing in different countries.

We provided more than 40 million rows of data from three completely different data streams. We also provided a template that ran a baseline solution, allowed connection to data sources, and permitted interaction with our comprehensive analytics platform.

Even though this wasn’t a requirement, nearly all the teams ended up using our template and analytics platform as they needed computational power to train their solutions. Thus, some data scientists were led to use our platform for the first time.

Solutions Created

It was impressive to see the ambition and commitment with which each team approached the task. Success required understanding a new task, developing a practical solution, while simultaneously dealing with novel data, in many cases learning new development practices, and making a compelling case for their work to a demanding jury. Every single team proposed a distinct solution and implemented a prototype producing meaningful results in allotted time. These accomplishments speak for the expertise of the data science community at TUI.

The proposed solutions variously tried to better understand customer queries, account for the purchase decisions of different segments, innovatively aggregated different searches within a given time window, and incorporated semi-structured data of hotel descriptions to build better models. They also proposed a variety of optimization schemes to align user and business interests.

Nearly all the teams produced results competitive with our baseline solution and many beat it on financial objectives. These solutions have provided practical insights and have spurred new developments in ranking projects.

Outcome

We conducted an anonymous survey to gauge participants’ feedback. The results exceeded our expectations as 100% of the participants said they would participate in a Hackathon again.

Takeaways

We learnt a lot during the process of organizing the hackathon, such as:

• Be very precise about the strategic objectives.
• Declare those objectives well in advance and in multiple venues to reinforce the message.
• Have clear and measurable criteria for evaluation of those objectives.
• A multi-dimensional criterion helps participants feel engaged.
• Hackathon must provide tangible, long-term value to the participants beyond the prizes or bragging rights.

Conclusion

Organizing the hackathon was a demanding task and required significant and sustained effort from our team, and it was absolutely worth it. Hackathon lead to a better understanding of our internal tech-tools, a greater alignment among our global data science community, and helped design a way forward on a critical business problem. We plan on making data science competitions a regular part of life at TUI.