Elasticsearch query to return all records — More Than 10,000 Documents?

The first article provides a helpful guide on how to use the “match_all” query in Elasticsearch to return all records in an index. However, maybe you want to see your document when your index has more than 10,000 documents. Below is an addition to my article that addresses this topic:

What Happens If Your Index Has More Than 10,000 Documents?

While the “match_all” query is a simple and effective way to retrieve all records in an Elasticsearch index, it’s important to consider the potential challenges when dealing with large indexes. Elasticsearch, by default, limits the number of documents returned in a single query to prevent excessive resource consumption. The default limit is set at 10,000 documents.

If your index contains more than 10,000 documents, running a “match_all” query as described earlier will not return all the documents in a single query. Instead, Elasticsearch will only return the first 10,000 documents. This is known as the “index.max_result_window” limit.

How to Retrieve More Than 10,000 Documents

If you need to retrieve more than 10,000 documents from an index, you have a few options:

1. Increase the “size” Parameter: As mentioned in the article, you can increase the “size” parameter in your query to specify the number of documents you want to retrieve. For instance, you can set it to 1000 to get the next 1,000 documents:

GET index_name/_search
{   
  "size": 1000,
  "query": {
    "match_all": {}
  }
}

Be cautious when setting a very high value for “size” as it can put a significant load on your Elasticsearch cluster.

Paginate search results | Elasticsearch Guide [8.10] | Elastic

2. Use Pagination: For more controlled retrieval of large result sets, you can implement pagination by using the “from” and “size” parameters. The “from” parameter specifies the starting point for your query, and the “size” parameter controls the number of documents to retrieve in each page. For example, to retrieve documents 12 at a time:

GET index_name/_search
{
  "from": 0,
  "size": 12,
  "query": {
    "match_all": {}
  }
}

Implementing pagination allows you to access all documents in smaller, manageable chunks.

Paginate search results | Elasticsearch Guide [8.10] | Elastic

3. Scroll API: If you need to retrieve a large number of documents efficiently, especially for processing or exporting data, you can use the Elasticsearch Scroll API. It provides a way to scroll through large result sets without the limitations of the “size” parameter. You initiate a scroll query, and Elasticsearch keeps a temporary snapshot of the search context. You can then scroll through the results batch by batch.

Scroll API | Elasticsearch Guide [8.10] | Elastic

4. Efficient Pagination with “search_after”
The “search_after” parameter offers a more performant alternative to “from” and “size” when dealing with large datasets. Sort Your Query: Ensure your query includes a sorting criterion, such as sorting by a date field for chronological results.

GET index_name/_search
{
 "query": { "match_all": {} },
 "sort": [ { "date_field": "asc" } ]
}

Initial Search: Use “search_after” with a starting value (typically “null”) to retrieve the first page of results.

GET index_name/_search
{
 "size": 10,
 "query": { "match_all": {} },
 "sort": [ { "date_field": "asc" } ],
 "search_after": [null]
}

Retrieving Subsequent Pages: Continue by using the values from the last document of the previous page in the “search_after” parameter.

GET index_name/_search
{
 "size": 10,
 "query": { "match_all": {} },
 "sort": [ { "date_field": "asc" } ],
 "search_after": [last_document_value]
}

“search_after” streamlines pagination for large datasets, making it ideal for applications that require efficient retrieval and processing, such as log analysis or data export tasks.

In summary, Elasticsearch’s “match_all” query is a powerful tool for retrieving records, but for indexes with more than 10,000 documents, you need to consider the limitations and choose an appropriate method to retrieve and manage your data effectively.