Making Location-Based Experimentation a part of our DNA
What can we learn about experiments on the ground from web-based experimentation?
“Our success at Amazon is a function of how many experiments do we run per year, per month, per week, per day” — Jeff Bezos
Experiments on the Web
The way that web analytics has progressed in the last decade has been fascinating, to say the least!
Web analytics tools have democratized user and product analytics and hence empowered marketers to track how their web apps are performing — things like where users are churning, what the funnel looks like, how to optimize the pricing page and so on.
Then, came the era of A/B testing tools. Once you knew something is not optimized, these experimentation tools made it extremely easy and quick to form a hypothesis, make a change, test it out on a set of users and track their behavior and over time automate some strategies.
This allowed marketers and designers to test every change, ranging from the size and color of a button to the language of a promotion!
This is the exact same philosophy that is preached in the startup world while developing products: Form a hypothesis, build an MVP and then see how users use it to iterate and evolve!
Experiments on the Ground
If we talk about experimentation and rapid iteration in the ground world, we have not even started thinking this way! We know so little about our real world and ground movement.
And given the fact that our “offline” world is so chaotic and fickle (protests, rain, local events, accidents, traffic) that gives us all the more reasons to learn which strategies work in which location and what time!
Web analytics helps us understand how every user behaves and what their tastes and preferences are!
But, understanding of our ground reality requires us to understand our users (demand), our ground partners (supply), along with the characteristic of the location (affluence, price elasticity type).
For example, let’s say there is a mismatch in the demand and supply at any point in time. It makes sense to drill down into what is happening behind the scenes:
- Is there a mismatch between the low valued customers and the supply?
- Is it because the area in which the partners are incentivized to go is a high traffic area?
- Is it because the partners are idle but they are still not accepting the orders?
- Is it because the store is loaded at the moment and cannot cater to more demand?
The ground reality is about how all the different dimensions of your business (that have a location and time component) interact with each other and in some cases with external events as well (weather, local events, traffic).
Time to Start Thinking “Spatially” about our Experiments
Let’s say we need to deploy a surge pricing module on the ground. Not only this surge has to be dynamic on time and location, but also on the user present in that location!
Uber’s surge pricing model works this way. Read more in case you are interested:
To give you an example, the way a surge of 3x would perform in a price-sensitive area would be completely different than how it performs in a more affluent area.
Before, even thinking of experimenting, automating or predicting, we need to first analyze what is going on and detect bottlenecks.
(i) Monitoring
Monitoring translates to tracking and learning what’s happening on the ground in real-time to make more tactical decisions.
Monitoring is really important for any kind of company where demand and supply in dynamic in a world which is so ever-changing in itself.
It enables us to get real-time feedback on what’s happening and detect anomalies and act on it!
For instance, let’s say the demand falls below 50% one night in area x or 60% of your riders are idle than the average in area y. It is in these moments that you need to have the constant inflow of information coming to you so that you can do something about it.
(ii) Detecting
Fixing your demand-supply gap in the instant, by asking drivers to move and or sending a marketing promotion, works in the short run.
But, if we know that every Thursday at 4 pm in area x, our users are dropping off because the ride is not available. You would have to tweak your strategy to make it fit for every Thursday evening.
Similarly, what are the areas where it takes more time to deliver an order on a weekend night? What are the areas where my partners spend the maximum amount of time stuck in traffic? Does it have any pattern with time, partner type or location?
While monitoring works well on tactical decisions, detecting or finding patterns on what’s breaking, again and again, is good for taking strategic decisions.
(iii) Profiling
Today, our imagination is limited to creating different kinds of user-profiles depending on what actions they perform. For example, “users who put the item in the cart but didn’t book” or “London users” or “music lovers”.
But how about creating location profiles depending on how demand and supply behave in that location?
For example, let’s say we create a profile called “Areas with high-value breakfast orders”. On top of these profiles, we can take action, run experiments and track what happens!
Affluent areas, office areas, residential areas, weekend party places are some examples of the profiles that you can create. If you know what people book rides every day during the weekday in the morning and evening, then it would probably be an office area.
(iv) Running Actions
The idea is when you have created a set of profiles, how can you run very contextualized strategies for them? In the web world, you send a different promotion to a very active user as compared to the promotion that you send to a hibernating user.
Similarly, it might not make sense to give a high amount of incentives to riders after 8 pm in an office area.
What kind of promotions, SLAs, surges, incentives, discounts, delivery charges work in different areas and at different for your business might be totally different.
(v) Testing and Iterating
The only way to learn is form intelligent hypotheses, test and see what sticks.
We also believe in giving the ability to the operations and supply teams to be able to tweak and iterate since they have the maximum context of how “their” areas behave. They should be able to add the knowledge present in their heads (commonly called “curb” data) to all the experiments and strategies.
And once they know for sure what works, automate it and make the learnings available across everyone in the organization!
Say Goodbye to the Good, Old Heatmaps
Why are we not fond of heatmaps?
Because heatmaps are just visual representation and they don’t quantify areas. Moreover, they defeat the purpose of these feedback loops.
Since you don’t know “what” is the area that is not performing well, it becomes really hard to make a decision. At Locale, we solve this by using a system of hexagonal grids.
Hundreads of companies that have increased their user acquisition, activation, conversion, and retention by doing web analytics successfully. At Locale, we are creating examples of companies who are improving their unit economics, cost per delivery, utilization by doing location analytics.
Similar Reads:
Why it’s time for a fresh, new product for real-time, location-based decisioning.
How we’re building our geospatial analytics product using first principles
If you want to delve further, check our website out or get in touch with me on LinkedIn or Twitter. Originally posted here.
