Let’s make ROI a fair game
By SRI², The search for the ideal multi-touch attribution solution
At Skyscanner, as with most companies today, we implement a Return on Investment(ROI) measure in investment decisions. Most of our everyday investment is in e-commerce marketing, and, naturally, we would like to measure the impact that each marketing campaign has on company performance. However, measuring this accurately is challenging: when we observe that our metrics are improving, where should we attribute that success to?
The traditional way of approaching this challenge, which has been adopted across the industry, is to use a method known as ‘Last touch attribution.’ This method simply says that a marketing campaign’s success is fully attributed to the last (or previous) touch point that a user had before doing some action that we care about. If the marketing touch point is not fortunate enough to be in the right place at the right time, it’s impact may not be seen at all.
At Skyscanner we recognize the importance of this and have started working towards a multi-touch attribution solution.
Begin with the end in mind: What do we aim for ?
In a traditional supervised learning experiment, the goal is to build a model that can predict a label or outcome by feeding an algorithm with millions of examples. However, we are trying to predict something that we cannot directly observe just like the quality of life, morale, or happiness.
We began by diving into the what has worked before, and found a variety of methods.These include Shapley value, Markov Graphs ; many of these looked suitable for our needs if we had a way to pick the best one. How could we measure which one of them was ideal for us?
How do we measure if the multi-touch solution is good enough?
At Skyscanner, we are working on a way to benchmark these different methods through a simulated marketing environment.
Based on Skyscanner’s historic data, we simulate marketing activity and set expectations on what should be the true value of various marketing touch points. The true value expectations are set by our marketing experts and our internal experimentation knowledge.
The simulated reality data will capture our expectations for:
- Distribution of touch points over user population
- Distribution of conversion over user population
- Distribution of various marketing touch points over all touchpoint
- Distribution of the marketing touch points over our users’ journey
- Order of channel exposure
These distribution properties can be captured from the Skyscanner data itself. However for the marketing touch point true values, in the absence of observable data, we use estimates. Hence, now we have a simulated true value to aim for and we can benchmark the different methods against each other to identify the best solution.
If our estimates of true values are spot on, it helps us predicts what actually happens in reality for Skyscanner. Which would be brilliant!
If our estimates are not spot on, the simulated marketing environment still adds a lot of value. It will still represent a multi-touch problem and enable us to identify which methodology is best to solve the problem in general.
It can be very challenging to get right, keeping Lean start up principles in mind, we aim to refine this solution iteratively as we learn more from our data, stakeholders and live test results.
Stress testing for Multi-touch Attribution
With the simulated marketing environment in place, we can generate stress testing scenarios:
- Non-Brand Search Engine Marketing(SEM) activity increase/decrease by 50%
- Stop Retargeting activity
- Set Retargeting activity to equal/more/less value as Acquisition marketing
- Increase/decrease traffic by 30 % overall
These are just some of the stress testing scenarios. This helps us to test the durability of the multi-touch model in plausible situations and the presence of temporal changes.
Hence, now we have an idea of the best methodology and ideal tunings to solve Multi-touch attribution.
We now apply the winning method to our live data.
CLV Attribution: From Multi-touch Attribution to Customer Lifetime Value
The aim is to eventually attribute Customer Lifetime Value(CLV) through Multi-touch attribution, to measure the impact our current marketing efforts will have on the company looking 12 months forward.
As we learn more we’ll share more, so stay tuned.
Want to know more?
We are presenting our research at the 12th Women in Machine Learning Workshop on 4th December 2017.
We will be elaborating on more research details of Multi-touch solution assumptions, methodologies, testing.
SEE the world with us
Many of our employees have had the opportunity to take advantage of our Skyscanner Employee Experience (SEE) — a self-funded, self-organized programme to work up to 30 days during a 24 month period, in some of our 10 global offices. There is also the opportunity to work for 15 days per year from their home country, if an employee is based in an office outside of the country they call home.
Like the sound of this? Look at our current Skyscanner Product Engineering job roles.
About the author
My name is Sri Sri Perangur, a Data Scientist working Skyscanner HQ in Edinburgh. I work with the growth side of the business on concepts such as CLV and Multi-touch as seen above. You can find more about my previous work here .
