Why we switched from Elasticsearch
In an ideal world, 100% of a software engineer’s time would be spent on developing or planning unique features for their company. Obviously, in reality, that is not the case. Developers have to fix bugs, maintain the infrastructure, write documentation, and develop the same boiler-plate code that has been written many times over by many other developers. The more boiler-plate code a developer has to write, the less time they are spending on features that are really going to push the company forward, or in the case of Appear Here, the less time we spend working on important features, the less we can help our website make ideas happen.
When I first joined Appear Here we were using Elasticsearch for our space search (space being the Appear Here word for the location a brand can lease with us). Elasticsearch is a great solution for search, but it simply does not work out of the box, it can require a lot of setup. To be able to use it properly, i.e. to provide a search we would consider satisfactory, the developer would have to have a strong understanding of how Elasticsearch works. The more time a developer spends learning the intricacies of Elasticsearch the less time they can spend on building features that can truly push the company forward. To avoid the “bus factor”, more than one developer on a team would have to get up to that standard. In a startup, time is critical, and the cost in time was simply too much.
We were using Elasticsearch for both parts of our space search. Firstly, for providing the autocomplete locations when a user was looking for spaces in a specific location, and secondly, for providing the spaces within a certain location when it was searched. The data for the autocomplete was gathered from a variety of sources and then indexed together. However, this did cause some problems, some data was repeated across multiple sources, meaning we had to de-duplicate it, and some data was missing, which meant we also had to provide custom data.
Overall, we were doing a lot of unnecessary work for a problem that had already been solved by many other companies.
The location data we had in our Elasticsearch did not contain any streets, such as Redchurch Street. We knew that key streets across London, Paris and New York had to be added to our autocomplete to assist our users finding them. This meant that in order to add these streets and maintain the existing way of things working we would have had to find a new location source for streets across these three cities and add them to our index. With the complicated setup we had, any change in one place seemed to have unexpected effects on our results. We were in a situation where any change would break something somewhere else in the search.
We had a few options, but we decided our best option was to simplify our search as much as possible. Our CTO, Andrew Thompson, mentioned that I should investigate services such as Algolia and SwiftType. Algolia provides search that is reliable, fast, and extremely easy to use. It was founded in 2012, and was part of Y Combinator’s 2014 class [1]. Swiftype is similar, it was also founded in 2012. In order to effectively find the spaces we want in an area we also needed a service which provides a geolocation search, which is exactly what Algolia provides. We set up a quick one week spike in order to test how easily we could replace our existing search functionality with Algolia. It only took a few hours to index all our spaces, and then to produce a demo application where the spaces were returned within the location requested. Unlike Elasticsearch, it only took a few hours for a single developer to understand how Algolia worked.
The ease of use of Algolia was perfect for us, freeing up time for the developers to focus on other things.
Elasticsearch ranks by giving various attributes weights which it then uses with relevance to calculate a score [2]. Whilst this feature can be very useful in certain circumstances, it does lead to some obscurification. We needed the ability to explain to various members of the company, such as our landlord team, why certain spaces were appearing in certain orders. Especially as we were only showing results within certain bounding boxes, the relevance of the words in the name or description of the space were far less important to us. Algolia’s ranking system is simpler and more transparent, we simply add custom properties to the ranking, and Algolia would then tie-break them against each other [3]. If we gave scores to each property, if both are the same, then it will move onto the next property, and so on, till the order of results is established. This allowed for a very simple ranking system that could be explained to anyone within the company, and allowed for quick and accurate decisions to be made on how to change how we rank results.
Elasticsearch can be very fast if setup correctly. There are a number of options for hosting it, such as AWS’ Elasticsearch service. At Appear Here we were not using it like that, we had a Rails application in front of it. This led to some unnecessary slowdown. Even if we did not use Elasticsearch the way we did, there is still infrastructure that needs maintaining, and people would still have to learn Elasticsearch in depth to be able to get the best performance. This was another advantage which led us to choose Algolia over Elasticsearch, Algolia guaranteed uptime with no additional work from us, once again, further freeing up time for development.
Whilst Algolia was able to provide an incredible replacement for our space search, we still had to find a solution for our location search. We had recently added Amplitude events to our search, which began to show that the majority of users were typically searching for cities. The assumption was made that smaller locations were more important, but some of the locations people expected to be searched only had a couple per month. This freed us up from needing a more custom approach to locations, and enabled us to choose a third party for location data. As we had just used Algolia for our space search, we looked at a service provided by Algolia called Algolia Places [4]. Whilst it seemed like a good application, there were a few desired features missing, for example, we wanted to limit the results to the countries we had offices in at the time, UK, France and USA. We decided we had to find another service providing location data.
This led us to look into the Google Places API. Google has been doing Maps for years, and their Places API was well tested. As desired, we were able to limit the location results within the three countries we needed at the time, and, based on which version of our site the user was using, we were able to boost the relevance of one country’s results over others. There were a few problems however, as can be the case when so many results can be provided.
One example of the problem with Google Places API was ‘Old Street’ in London was showing up the same for both the street and the station. This was a big problem to us as Old Street station contains many important spaces which needed to show high on the space results. Secondly, having the same location suggestion appear twice did not provide a good user experience. We had to have a simple workaround where we filtered out some results based on their PlaceID, in the case of Old Street, we filtered out the street.
A further issue with Google Places API was caused by the bounding boxes provided for the selected location. Not all areas fit neatly into a rectangle, the best example is Brooklyn in New York. When a user searches ‘Brooklyn’ the box around it meant that a large number of spaces within Manhattan were also showing up and being shown as more relevant as spaces within Brooklyn. Again, an approach was used where we had to replace the bounding box provided by Google with smaller bounding boxes which covered Brooklyn more effectively. We then passed on these smaller bounding boxes to Algolia, which is able to do searches within multiple bounding boxes.
Besides these few customisations we had to apply on top of Google Places API, we were very happy with how it worked for us. We gained a reliable and well trusted set of data, whilst having to do minimum work to perfect it to make it work for us in the way we desired. When combined with Algolia, it made for a powerful and fast search which was able to effectively provide users with the results they were looking for.
For a company the size of Appear Here, Algolia has been an amazing tool which we would recommend to anyone in a similar situation. We were particularly impressed by the ability to rank results within a location. Beyond just spaces, as we continued to develop our site we found many uses for it within the admin part of our site. The developers who used it were able to very quickly understand it, and implement a new search within a very short time-frame. With search solved for Appear Here, the developers were free to work on the features that could really help our site make ideas happen.
[2] https://www.elastic.co/guide/en/elasticsearch/guide/current/relevance-intro.html
[3] https://www.algolia.com/doc/guides/managing-results/must-do/custom-ranking/#custom-ranking
