Elasticsearch: keyword analyzer vs english analyzer

Deciding over which analyzer to use while indexing a field in ElasticSearch can really spin your head around sometimes. Today I would like to discuss the two analyzers which earlier I found most confusing and left me thinking what is the difference between the two and in which situation I should use what.

For instance: There is a table named JOB which has many categories and I would like to index it. So which one should I choose?

This one:

indexes :categories, type: 'keyword'

Or this one:

indexes :categories, type: 'text', analyzer: :english

Super confusing right?

Well the answer is that it depends on the situation in which you would be using the particular index.

The first analyzer is a keyword analyzer does exact match of the query. So, for instance, if you’ll search "Project management " as a category, it will not show up in searches for "project" or "management" on their own. But if we want case insensitivity in our result and want our result to also return "project management", then we need to index it like this:

indexes :categories, type: 'keyword', normalizer: :lowercase

On the other hand, the english analyzer is a text analyzer, hence it will break the text into individual words and it will show up with “project” or “management” on their own. In addition, using the english analyzer means that plurals or verb tenses are ignored. So “manager” or “projects” would also find “project management”.

Therefore, I tend to use keywords for things like categories because often the user selects the category from a list, so we want an exact match. If the categories are used for filters, then you can go straight for keyword. To put in other words, analyzer should depend on whether a person would be typing and search something or choosing to search from a predefined list. If user has to enter the things that has to be searched, use english analyzer, but if he has to choose from a list, then use keyword analyzer.

I hope you found it helpful. :)

Originally published at https://blazarblogs.wordpress.com on June 5, 2020.