The search is dead, long live the Search!
A few months ago our engineering team at badi had to carry out a major refactoring of a critical component on our Rails application — the search engine — to improve the performance and to enable more advanced features. Given the key role of the service and its heavy use, we had to make sure no users were negatively impacted while rolling out such complex change.
The strategy taken was to iterate over small steps to assess at each stage whether the refactoring was aligned with the expectations. This agile approach gave us good visibility on the status of the refactoring, the ability to react to possible problems, and a better estimation of the overall project.
Since all decisions at badi are data-driven — which also applies to us, the engineering team — we decided to approach the refactoring by performing empirical observations and measuring the success based on defined metrics. A key part of this operation was to generate good data quality to input in the search engine in order to have relevant results to compare.
Objective 🚧
Search is the default landing experience for most users, and relying on the SQL queries was no longer an option. Our traffic increase quickly led to it becoming our main bottleneck. We needed a change.
After evaluating alternatives, we decided to offload the database. We opted for Elasticsearch, a high performance system optimised for this task. Now that the replacement was chosen, the fun part started!
Strategy 🤓
Having defined the process, we started evaluating a few different tools that would enable our experimental approach without losing the flexibility needed for our iterations. After some research we found an open source gem released by GitHub, Scientist.
Scientist allows you to create an experiment on top of old code and new logic. This experiment executes both paths at runtime, collects metrics and continues sending the results from the legacy code to the users. Most importantly, we could use real data to evaluate the new infrastructure results.
We found the right tool for the job!
We extended Scientist’s custom behaviour to track the following comparison metrics:
- Response time: time taken by the component to execute the logic
- Number of matching results: return a boolean depending on whether the result of both components is exactly the same
- Percentage of matches: returns the percentage of similar items between both responses
We combined this with our New Relic integration to collect custom events and metrics. This helped us create relevant dashboards used to monitor the process.
Alright! We had all the infrastructure in place to start with the refactoring!
Well… not quite. As we said above, the search component includes a complex piece of logic based on several filters that could be called independently or by any permutation of them. Even with a good set of data to feed the search, we would still need the Elasticsearch component to be fully implemented in order to be able to compare the results.
What we wanted was to ensure that the result of each filter was aligned with the expected behaviour before the search component was completed. That required having more control of which filter to enable while performing the measurement.
Since our application was already using a feature flag system to perform A/B testing, we decided to apply the same enabling/disabling procedure to our search filters — every time a new filter was completed, we assigned a flag and ran the experiment.
The traffic from the search component would then be funnelled into the experiment only for the filters that were enabled. In this way we were sure to track the metrics only for the expected request. Adding this granularity to the process allowed us to validate our results with more confidence and assess the refactoring status at each iteration.
The Experiment 🔬
We were finally ready to start!
We isolated all the filters used by the old query and started to implement them one by one following the process defined below:
Every time a new filter was implemented for the Elasticsearch component, we would create a new feature flag to be able to enable it. First we tested it in isolation, then in conjunction with the filters we had already implemented. If the result was satisfactory, we moved on to the next filter; if not, we would investigate what the problem was until we reached the expected result.
Above is an example of the metrics gathered during our iterations. After running the experiment for some time we could compare the results and check whether the filter was correct or not.
We ran this process until we covered all the filters present in the database component and validated the results with the data gathered from the metrics. From the chart on the dashboard we were happy to assess that the matching results were above the threshold level we had set and the overall response time for the new component was twice as fast.
The threshold was defined because we were aware that one of the most used filters, the bounding box, would have slightly different results because of the algorithm between the two components. This led the metrics to have some occasional mismatch.
Given this outcome, the final step of the process was to clean up the code, removing the experiment framework to allow the search service to use the Elasticsearch adapter exclusively.
Conclusion ⚖️
Following this approach, we started our refactoring with some overhead, needed to set up the framework for the experiment. This paid off from the very beginning, in fact, we saw an enormous advantage of delivering on fast iterations and the ability to be reactive in case of problems. Another win was setting measurable expectations for each iteration, which simplifies the evaluation process intended to let us decide whether the search was performing as expected.
If you are interested in what we do at badi you can look at the current job openings or contact us directly.
