How a Single Search Experiment Generates Million-Dollar Impact for Indonesia
Search Engine in Bukalapak
One of the key features in Bukalapak is its search engine which helps the customer to find relevant products from Pelapak (sellers) who sell their product in Bukalapak. Given that Pelapak registered in Bukalapak are coming from a broad range of business unit scale from newcomers to heavyweight contenders, we want to facilitate and help those Pelapak newcomers product have same discoverability with the old comers (heavyweight contenders). Bukalapak then build promoted push feature that can help newcomers possibility to have greater discoverability on their product when potential customers try to find a particular product they need by appearing at the first page of relevant result.
Promoted push feature basically uses auction principle but additionally, it combines auction bid and product quality scoring to decide which product appears at the top section of the search results. One of the key challenges in building this feature is balancing between monetization factor and quality factors in order to have effective CPC and fast ROI for the Pelapak sellers that use the feature.
Core Search Squad at Bukalapak
Our Core Search squad consists of several engineers, a data scientist, and a software architect. (me!) Due to the potentially high impact of experiments at our squad, we are fortunate to have Putri, a data scientist with PhD in predictive modelling and biostatistics, as part of our team. She has been invaluable in driving forward our experiments and analysis.
Our approach to improving promoted push feature effectiveness
When dealing with monetization, we understand that we need to do it carefully due to its sensitive nature. Prior to improving the feature, we first defined our approach to understand the landscape & environment and what might be a relevant factor can be included as part of auction effectiveness factors.
- Understanding seller and buyer profiles
Search engine plays a great role in connecting potential buyers and sellers, a bridge for sellers and buyers to fulfil their needs. The search result itself is one of the media to connect the buyer with seller’s product. For a better search result, we should understand/recognize who our buyers and who our sellers are- as both of them are our customers. Having understood and recognized the characteristics on both sides by analyzing their population and distribution in each cluster/profile, then we built building appropriate relevancies strategy that can be a win-win solution for both sides (sellers and buyers).
2. Evaluating auction effectiveness factor
One of four factors that might increase effectiveness after “Recognize” is “Relevance”. So we ask ourselves how we can make search results more relevant. Making more relevant while adding an auction bid factor but can increase click and conversion rate without any disruption is challenging attempts. What we have done is taking one step back and evaluating what makes our search result more relevance in the organic section. We evaluating it by selecting and tuning relevant signals/factors and it's weight contribution that fit in the auction section. Additionally, we take extra miles — through the help of our data scientist — applying machine learning analysis, and revealed the unrevealed relevant signals and its weight. This is the point where human and machine complement each other — filling “the missing middle’.
3. Communicate Approach to Stakeholder
After preparing our approach and validating our machine learning models, we then communicate them to our stakeholder on how we can improve the auction section both from seller side and buyer side including our implementation and rollout plan & strategy. The communication was done in both-ways discussion in a very constructive spirit — both business and the technical team preparing what needs to be done in order to make it success it two phases, the experimental phase, and the live rollout phase.
4. Implementation, AB Test, and Evaluation
Once our approach, model, and strategy agreed by the stakeholder we then right away implement the improvement. At Bukalapak, each and every changing should pass A/B test before rolling out it to the production — as such so we did and prepare A/B test with two variants (incumbent strategy, and new strategy). The Purpose of the AB Test is to make sure that a new strategy would outperform the incumbent strategy. If the result during the experiment said vice versa, then we can reduce or isolate the loss. AB Test was run about 2 weeks duration with two rolls out iterations:
- Iteration -1: 50 : 50 traffic
- Iteration-2: 20: 80 traffic
Both iteration results show that the new strategy outperformed the incumbent strategy with significant double-digit percentage increase on overall conversion rate, an additional US$1 million increase in annual revenue.
More importantly, the new strategy help drives tens of millions of USD in additional sales to our small sellers, bringing more impact to their livelihood. Overall, we believe this new strategy means that sellers can more effectively use their auction bid to increase their sales conversion.
If you are enthusiasts on composing technical strategic delivery to improve Bukalapak search engine, visit our career site (https://careers.bukalapak.com) and let’s join us.
