If you can’t measure it, don’t do it
I have been told this quite a few times by my boss and I have to say I have adopted it entirely now. I’m going to tell you why.
KPIs, metrics, OKRs, ROI, everything today is about numbers. And for a good reason, how can you prove the investment is worth it?
Why should you care?
Whether you are in a start-up or scale-up (this is where TheFork stands), there is a series of important steps before you are able to aggregate data and make decisions:
- We have a fully operational ELK stack for production monitoring
- We have built a datalake, centralized location where we gather and consolidate production data
- We have a long list of JIRA issues and various metrics about how long it takes to resolve a given issue
- We have many insights from our partners (the restaurants) and reviews/comments from our customers (you and me)
- Regular traffic monitoring tools such as Google Analytics or more UX oriented ones such as Hotjar
Today data is here, it’s now time to benefit from it.
Leverage data to build and retrospect
No matter the stage your company is in, when it comes to costs, you always need to make the right decision or at least feel this is the best direction to follow.
Building
At TheFork, we define annual company OKRs that then break down into quarterly OKRs and eventually 8 weeks cycles (6 for implementation and 2 for cool down period: control over tech debt and shaping up next cycle).
Picking the right topics to focus our teams on is key to succeed in our objectives. Each cycle, Product & Engineering teams gather and brainstorm upon business metrics to make educated choices about features having the greatest impact for our customers.
The successful example of TFM3 search experience
Over the time, the product team discovered several issues with the search feature available for restaurants to find customers in their B2B application.
Collecting data, they have been able to state that:
- We had two outages in less than 6 months because of this feature
- Response time from the search service (based on SolR) was too high (>500ms and up to 1 or 2 second(s) for the autocompletion)
- That led to a serious increase of customers duplication (it took too much time to retrieve the customers list, then it was quicker for the restaurant to create a new customer from scratch)
- SolR index size is dramatically larger than expected (~250 Go vs 75 Go target) and doesn’t scale right
The proper response to that assessment was to rebuild the feature, leveraging Elasticsearch technology.
Having existing metrics in mind, we were able to define OKRs to track this project.
All along the cycle, we have been using the data to project response time, estimated growth, index size. We will then be able to measure the improvement and confirm the direction we have chosen was the right choice.
The -less successful- example of Custom Group Preset Menu
Before TFM3, we had… TFM2, this giant monolith but with some really cool features our restaurant are using everyday. There was this feature we didn’t have yet in TFM3: Custom Group Preset Menu.
Goal is simple, provide a unique preset menu for a group reservation, the restaurant will propose a list of dishes, different from the regular-available-for-everyone preset-menu.
So we took the bet and implemented this in TFM3. This was 2019.
Looking at data today, was it worth the investment?
If you take Sweden for instance, the country that uses the most this feature, it represents 0.01% of total bookings. I guess we have an answer.
However…
While making sure you’re not wasting money on not-that-useful implementations, it’s also important to keep time to try and fail, innovate and follow your gut feeling from time to time.
Also, sometimes you won’t have the data you need so you will make assumptions, that’s a bet after all. Once your project is rolled out, you will have data to transform your assumption into fact.
To help confirming as soon as possible your assumption, I’ll recommend prototyping first, measure and then take time to properly implement.
Final thought
If you have the necessary means, having data at your disposal will definitely help you get confidence when it comes to make the right call.
More than ever today, data is key, it’s then really worth taking the time to invest in data infrastructure and teams to help you succeed in your business.
