Understanding the Elasticsearch Query DSL

Steven Li
Steven Li
Apr 26 · 5 min read

What This Tutorial Is About

Elasticsearch is the go-to search engine these days, but its Query DSL does have a steep learning curve. When I first started writing Elasticsearch queries, I could string together something that worked through a combination of the Elastic.co docs and Stack Overflow, but I didn’t fully understand the underlying concepts behind the syntax.

This tutorial aims to be what I wish I first read. It assumes the reader is familiar with basic Elasticsearch concepts, can write simple queries, and understands boolean logic. It aims to provide the reader with a firm conceptual basis of Elasticsearch before trying to understanding the syntax.

Filtering Exact Values vs Full Text Analyzed Search

An over-arching theme in Elasticsearch is that all queries can be classified into three types:

1. Filtering by exact values
2. Searching on analyzed text
3. A combination of the two

Every document field can be classified either as an exact values or analyzed text (also called full text). Exact values are fields like user_id, date, email_addresses, etc. Analyzed text is text data like product_description or email_body. As the name implies, this text data has been analyzed (more on this later). It is often in a human natural language but not necessarily.

Querying documents can be done by specifying filters over exact values. In these cases, the question of whether the document gets returned is a binary yes or no. For example, is the document’s user_id equal to 174517 ? Is the document’s created_at date within the range of the last month?

On the other hand, querying documents by searching analyzed text returns results based on relevance. A document is returned not by a yes/no criteria, but by how relevant it is. For example, if a document’s analyzed text includes Johnny Depp, it should also be returned in searches for John Depp or Johnnie Depp. A search for cook should also return results for cooking and cooked.

From this behavior, we can deduce that searching by analyzed text is a highly complex operation and involves different analyzer packages depending on the type of text data. For example, some analyzer packages are language specific which are used on analyze text in certain language. The default analyzer package is the standard analyzer which splits text by word boundaries, lowercases and removes punctuation. Because searching by analyzed text is so much more complicated than filtering by exact values, it is much less performant than just filtering by exact values. Note, we will call searching by analyzed text as analyzed search for short.

The Query DSL

Elasticsearch queries are comprised of one or many query clauses. Query clauses can be combined to create other query clauses, called compound query clauses. All query clauses have either one of these two formats:

{
QUERY_CLAUSE: {
ARGUMENT: VALUE,
ARGUMENT: VALUE,...
}
}
{
QUERY_CLAUSE: {
FIELD_NAME: {
ARGUMENT: VALUE,
ARGUMENT: VALUE,...
}
}
}

The syntax-rule is that query clauses can be repeatedly nested inside other query clauses

{
QUERY_CLAUSE {
QUERY_CLAUSE: {
QUERY_CLAUSE: {
QUERY_CLAUSE: {
ARGUMENT: VALUE,
ARGUMENT: VALUE,...
}
}
}
}
}

Here are some common query clauses.

Match Query Clause

The match query clause is the most generic and commonly used query clause. It’s fairly smart in that when it’s run on a analyzed text field, it performs an analyzed search on the text. When it is run on an exact value field, it performs a filter.

In the example below, the first query clause will perform an analyzed search since description is an analyzed text field. While the second 2 queries are filters over exact value fields.

{ "match": { "description": "Fourier analysis signals processing" }}
{ "match": { "date": "2014-09-01" }}
{ "match": { "visible": true }}

The Match All Query Clause

The match all query clause returns all documents. It’s analogous to SELECT * FROM table in SQL.

{ "match_all": {} }

Term/Terms Query Clause

The term and terms query clauses are used to filter by a exact value fields by single or multiple values, respectively. In the case of multiple values, the logical connection is OR .

For example, the first query finds all documents with the tag “math”. The second query finds all documents with the tags “math” or “statistics”.

{ "term": { "tag": "math" }}
{ "terms": { "tag": ["math", "statistics"] }}

Multi Match Query Clause

The multi match query clause is a match query that is run across multiple fields instead of just one.

{
"multi_match": {
"query": "probability theory",
"fields": ["title", "body"]
}
}

Exists and Missing Filters Query Clause

The exists filter checks that documents have a value at a specified field. The missing filter checks that documents do not have have a value at a specified field. They are analogous to SQL’s IS NULL and IS NOT NULL clauses.

{
"exists" : {
"field" : "title"
}
}

and

{
"missing" : {
"field" : "title"
}
}

Range Filter Query Clause

The range filter query clause is used to filter number and date fields in ranges, using the operators gt gte lt lte short for greater_than greater_than_or_equal less_than and less_than_or_equal , respectively.

{ "range" : { "age" : { "gt" : 30 } } }{ 
"range": {
"born" : {
"gte": "01/01/2012",
"lte": "2013",
"format": "dd/MM/yyyy||yyyy"
}
}
}

Bool Query Clause

Query clauses that are built from other query clauses are called compound query clauses. Note that compound query clauses can also be comprised of other compound query clauses, allowing for multi-layer nesting.

The bool query clause is an example of a compound query clause, as it is used to combine multiple query clauses using boolean operators. The three supported boolean operators are must must_not and should , which correspond to AND , NOT , and OR , respectively.

For example, suppose we have an index on the posts of a popular social media site. Here is a query to find all posts about math that is not probability, where it is either unread or has been favorited.

{
"bool": {
"must": { "term": { "tag": "math" }},
"must_not": { "term": { "tag": "probability" }},
"should": [
{ "term": { "favorite": true }},
{ "term": { "unread": true }}
]
}
}

Combining Analyzed Search With Filters

We have been talking about exact field filters and analyzed search in separate contexts, but in real world applications, we often want to combine the two. We combine analyzed search and exact field filters using the filtered clause.

For example, suppose we have an index on the posts of a popular web forum on mathematics. Here is a query to find all posts by performing an analyzed search for “Probability Theory” but we only want posts with 20 or more upvotes and not those with that tag “frequentist”.

{
"filtered": {
"query": { "match": { "body": "Probability Theory" }},
"filter": {
"bool": {
"must": {
"range": { "upvotes" : { "gt" : 20 } }
},
"must_not": { "term": { "tag": "frequentist" } }
}
}
}
}

Summary

The conceptual backdrop of the Elasticsearch query DSL is this dichotomy of filtering documents vs searching through analyzed text. Hope ya’ll found this helpful.

Feel free to leave comments, questions, suggestions, corrections below. — S

Steven Li

Written by

Steven Li

Writing About Rails, React, Web Application Technology, Databases, and Software Engineering

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade