Restoring Elasticsearch Snapshots: Protecting Your Data Across Environments

Restoring Elasticsearch Snapshots: Protecting Your Data Across Environments

In our previous blog post, we discussed how to set up automated snapshots of your Elasticsearch data to Amazon S3, ensuring that your critical information is regularly backed up and protected. But what happens if you need to restore that data to a different Elasticsearch cluster, perhaps in a new environment or on a separate server?

In this follow-up post, we’ll guide you through the process of restoring an Elasticsearch snapshot, allowing you to quickly and reliably recover your data in the event of an emergency or during a migration.

Prerequisites

Before we dive in, make sure you have the following in place:

1. Elasticsearch Cluster: You’ll need to have a running Elasticsearch cluster where you want to restore the snapshot.
2. Elasticsearch S3 Repository Plugin: As with the backup process, you’ll need to install the Elasticsearch S3 repository plugin on the target Elasticsearch cluster.
3. Snapshot Repository Configuration: You’ll need to configure the same S3 bucket and repository settings on the target Elasticsearch cluster as you did for the backup.

Restoring an Elasticsearch Snapshot

Install the S3 Repository Plugin:

cd /usr/share/elasticsearch
sudo bin/elasticsearch-plugin install repository-s3

Verify the Plugin Installation:

sudo ES_PATH_CONF=/etc/elasticsearch /usr/share/elasticsearch/bin/elasticsearch-plugin list

You should see the repository-s3 plugin in the output.

discovery-ec2
repository-s3

Register the Snapshot Repository:

3.Register snapshot repository per index with base_path (prefix) in S3 bucket for ES cluster (one-time operation)

• Bucket name = elg-testbucket
• base_path(profix)= elgbackup/
• snapshot repository = snapshot-s5-repo

curl -X PUT "localhost:9200/_snapshot/snapshot-s5-repo?pretty" -H 'Content-Type: application/json' -d'
{
  "type": "s3",
  "settings": {
    "bucket": "elg-testbucket",
    "region": "us-east-1",
    "base_path": "elgbackup/",
    "max_snapshot_bytes_per_sec": "500mb",
    "max_restore_bytes_per_sec": "1gb"
  }
}
'

1. Make sure the bucket name, region, and base path match the settings you used for the backup.
2. Check the Current Indices:

curl -X GET "localhost:9200/_cat/indices?v"

This will show you the current indices in your Elasticsearch cluster.

Delete All Indices (Optional):

curl -X DELETE "localhost:9200/_all"

If you want to completely restore your Elasticsearch cluster from the snapshot, you can delete all the existing indices first.

1. Restore the Snapshot:

List the snapshot you want to restore.

curl -X GET "http://localhost:9200/_snapshot/Daily-Backup/_all?pretty"

or

curl -X GET "localhost:9200/_cat/snapshots/Daily-Backup?v"

curl -X POST "localhost:9200/_snapshot/<Repo-Name>/<Snapshot-Name>/_restore" -H "Content-Type: application/json" -d '{
  "indices": "*,-.geoip_databases,-.ds-.logs-deprecation.elasticsearch-default-2024.03.20-000001",
  "ignore_unavailable": true,
  "include_global_state": false,
  "rename_pattern": "(.+)",
  "rename_replacement": "restored_$1_'$(date +%s)'"
}'

1. Verify the Restored Indices:

curl -X GET "localhost:9200/_cat/indices?v"

You should see the restored indices in the output.

Check the progress of restoring the snapshot.

curl -X GET "localhost:9200/_cat/recovery?v"

Note:

After the latest backup, you want to restore the newer snapshot then delete the current report and re-register it your new snapshot will be available

By following these steps, you can restore your Elasticsearch data from a snapshot stored in an Amazon S3 bucket to a new Elasticsearch cluster or server. This ensures that your data is always available and can be quickly recovered in the event of a disaster or during a migration.

Remember, it’s crucial to regularly test your restore process to make sure it works as expected and that you can successfully recover your data when needed. Keeping your Elasticsearch data safe and accessible is a key part of maintaining a robust and reliable system.