Searching for a Search Engine
Choosing the right tool to power the searches in your application.
Many applications need to provide a search function that is both fast and powerful. In the beginning, you can keep things simple and leverage your main database for that. Relational databases provide many operations you can use for searching — from basic pattern matching with the LIKE operator to full-text indexing. This approach helps to keep operational complexity low, as there is no need to manage a separate database and synchronize data across different systems. Depending on your requirements, this might be all you need.
However, as your application scales, there may come a point where the performance or features of this system become limiting. This signals the need to explore alternative solutions, such as dedicated search engines. These engines are highly optimized for searching through large amounts of data, potentially delivering significantly faster search results. In addition, they also offer other benefits, such as fuzzy search, which makes queries typo-tolerant, and give you more control over how you search and prioritize the results.
This begs the question of which search engine to use. Due to the number of options available, it can be quite a daunting decision to make. As we recently went through the same process at Scoro, I want to share some of the options you can consider if you ever find yourself in a similar situation.
Elasticsearch
Elasticsearch is a search engine built on the Apache Lucene library. With its initial release dating back to 2010, it is one of the most mature options in this list.
Elasticsearch can be freely self-hosted, but if you prefer a managed service, paid SaaS options are also available. It is very flexible, providing many options to set up your search functionality and fine-tune it for the best possible performance. However, this great flexibility comes with a steeper learning curve, and mastering Elasticsearch can prove to be challenging.
In 2021, Elasticsearch became the center of controversy, when it transitioned away from Apache 2.0 license to be dual licensed under the more restrictive SSPL and the Elastic License. While the source code for Elasticsearch is still available, these new licenses are perceived by many as a departure from traditional open-source principles.
OpenSearch
If the license change is a deal-breaker for you but you do like what Elasticsearch has to offer, look at OpenSearch. It is a fork of Elasticsearch created by Amazon.
OpenSearch continues to be released under the Apache 2.0 license and aims to be more open to contributions from the community. Today, both projects are under active development, but there seems to be a little more activity in ElasticSearch’s Github repository.
Apache Solr
Another open-source option built on Apache Lucene library is Apache Solr. In fact, both Solr and Lucene were developed under the same project for a long time before being separated in 2021.
It is by far the oldest solution in this list, dating back to 2004. Due to that, it has a proven track record, having been used in many projects over the years. Just like Elasticsearch, it is really flexible but can be even more complex to set up and configure. These days it seems to have lost some of its popularity to the newer contenders in this list.
TypeSense
TypeSense is another open-source option that you can either self-host or use the SaaS option provided by the company. It was released in 2015 and is written in C++, so compared to the previous Java-based options, it is easier to deploy as it’s just a single binary.
TypeSense is designed with user-friendliness in mind, offering sensible defaults and some configuration options to fine-tune search results. It achieves great performance by retaining all data in memory during runtime. However, it’s important to acknowledge that this makes the available RAM the limiting factor when it comes to the number of documents you can store. Therefore, it is crucial to carefully consider your data size and server resources when implementing TypeSense.
Algolia
Algolia is a proprietary search engine initially released in 2012. It is exclusively available as a SaaS solution and self-hosting is not available.
Algolia claims to be easy to set up and deliver exceptional performance. They provide a free tier limited to 10,000 requests per month and storage of 1 million records if you wish to try it out. If you are looking for a managed solution, it is definitely worth considering.
MeiliSearch
Wrapping up our list is the newest addition, MeiliSearch, introduced in 2018 and written in Rust. Just like TypeSense, it compiles down to just a single native binary.
MeiliSearch draws inspiration from Algolia and aims to deliver comparable features and performance. Unlike Algolia, MeiliSearch is an open-source solution and can be self-hosted, but paid SaaS options are also available.
In conclusion, there are many great search engines to choose from. Which one is right for you, depends on your specific needs and preferences. Whether it’s one of the options listed here or something else, each has its strengths and potential drawbacks. We encourage you to evaluate these options thoroughly and choose the one that aligns best with your requirements.
At Scoro, after researching our options, prototyping, and weighing the pros and cons of each, we decided to go with Elasticsearch. There were many factors that pushed us in that direction:
- We needed the flexibility to accommodate all the different use cases in Scoro, ranging from global search to drop-down lists that suggest items based on user input. These use cases come with different requirements and Elasticsearch provides us with all the tools needed to fine-tune the search experience for all of them.
- During our initial benchmarking, Elasticsearch demonstrated performance capabilities that met our needs.
- We have been using ELK stack for logging, so we already have in-house knowledge about running Elasticsearch clusters.
- Elasticsearch is a widely used and mature solution.
All of this gave us enough confidence to move forward with the decision.
If you’re passionate about working with exciting technologies and want to help us implement them, check out our open positions at https://www.scoro.com/careers/.
