Elasticsearch Snapshot and Restore Feature

Photo by Patrick Hendry on Unsplash

You perform the snapshot process if you want to take a backup in Elasticsearch. Today, We will focus on how to backup and restore Elasticsearch using snapshots.

Snapshot is an official feature of Elasticsearch with a Basic Plan (free to use).

We may take a snapshot with Kibana UI, Elasticsearch API, or scripts. In this document, We will use Kibana UI to prepare to take the snapshot process. But the Elasticsearch API has the same parameters, and We will share a sample API request.

IMPORTANT!

• A snapshot of an index created in 6.x can be restored to 7.x.
• A snapshot of an index created in 5.x can be restored to 6.x.
• A snapshot of an index created in 2.x can be restored to 5.x.
• A snapshot of an index created in 1.x can be restored to 2.x.

On the Kibana side, we may access, manage, and configure snapshots with the ‘Management’ > ‘Snapshot and Restore’ section.

Snapshot and Restore on Kibana

But first, we must create a repository for creating snapshots. We may create repositories with the ‘Repositories’ section on the ‘Snapshot and Restore’ page.

Click the ‘Register a repository’ button to create a repository. When we click that button, this section will be shown;

Create Repository

Repository Types:

• Shared File System: This option uses the machine’s file system and stores snapshots on the file system.
• Read-only URL: This option wants a URL to store snapshots. This URL might be AWS, Azure, Google Cloud-like systems, or a git repository.

IMPORTANT! Before setup the Shared File System, we should create the shared folder as an NFS server within cluster nodes.

Setting up the shared folder as an NFS server

Step 1: Install NFS for all cluster nodes (masters and data nodes)

#check nfs service installed
rpm -qa | grep nfs-utils

#install nfs service
yum install nfs-utils

Step 2: Create a folder where the snapshots will be stored (in this case we defined one of the Cold VMs)

# on Cold-S1 node
mkdir -p /mnt/data/sharing/es_backup
chmod -R 755 /mnt/data/sharing/es_backup
chown -R elasticsearch:elasticsearch /mnt/data/sharing/es_backup

Step 3: Edit the exports file and add below the line (only on the Cold-S1 node)

vi /etc/exports
/mnt/data/sharing/es_backup *(rw,sync,no_subtree_check)

Step 4: Reload the exports file (only on the Cold-S1 node)

exportfs -a

Step 5: Start NFS service (masters and data nodes)

systemctl enable rpcbind
systemctl enable nfslock => this may not be.
systemctl enable nfs-server

systemctl start rpcbind
systemctl start nfslock => this may not be.
systemctl start nfs-server

Step 6: Create a folder where we will mount the shared folder (masters and data nodes)

mkdir -p /mnt/data/es_backup

Step 7: Mount the shared folder (masters and data nodes, Cold-S1 node IP Address)

mount -t nfs ip_address:/mnt/data/sharing/es_backup /mnt/data/es_backup

Note: If You have NFS no route host, resolve is

firewall-cmd --permanent --add-port=111/tcp ... etc for the related ports.

Step 8: Permanent NFS mounting (We don’t apply this way, just for knowledge)

vi /etc/fstab
10.236.8.213:/mnt/data/sharing/es_backup /mnt/data/es_backup nfs defaults 0 0

Step 9: Add path.repo line in elasticsearch.yml(masters and data nodes)

vi /etc/elasticsearch/elasticsearch.yml
path.repo: /mnt/data/es_backup

Step10: Restart Elasticsearch service (masters and data nodes)

systemctl restart elasticsearch

Return to Kibana Side.

Shared File System

• File System Location (Required): This parameter, is a folder name of the stored data. This will be created inside the Elasticsearch directory.
• Snapshot Compression: Turns on compression of the snapshot files. Compression is applied only to metadata files (index mapping and settings). Data files are not compressed. Defaults to true.
• Chunk Size: Big files can be broken down into chunks during snapshotting if needed. Specify the chunk size as a value and unit, for example, 1GB, 10MB, 5KB, 500B. Defaults to null (unlimited chunk size).
• Max Snapshot Bytes Per Second: Throttles per node restore rate. Defaults to 40MB per second.
• Max Restore Bytes Per Second: Throttles per node snapshot rate. Defaults to 40MB per second.
• Read-Only: Makes repository read-only. Defaults to false.

After the configuration, the repository will be created. But this process will not be enough for taking snapshots. We must create a ‘Policy’ to complete this process.

Create a Policy

Policies, a process configuration of how snapshots will create.

Create Policy

• Policy Name (Required): Unique name of the Policy.
• Snapshot Name (Required): The name is automatically assigned to each snapshot created by the policy. This value supports the same date math supported in index names. To prevent conflicting snapshot names, a UUID is automatically appended to each snapshot name.
• Repository (Required): Repository used to store snapshots created by this policy. This repository must exist prior to the policy’s creation.
• Schedule (Required): Periodic or absolute schedule at which the policy creates snapshots and deletes expired snapshots. Schedule changes to existing policies are applied immediately.

Snapshot Settings

• Indices: Array of index names or wildcard pattern of index names included in snapshots.
• Ignore Unavailable Indices: If true, missing indices do not cause snapshot creation to fail and return an error. Defaults to false.
• Allow Partial Indices: Allows snapshots of indices with primary shards that are unavailable. Otherwise, the entire snapshot will fail.
• Include Global State: If true, cluster states are included in snapshots. Defaults to false.

Snapshot Retention

• Expiration: The time period after which a snapshot is considered expired and eligible for deletion.
• Snapshots to Retain:
• Minimum Count: Minimum number of snapshots to retain, even if the snapshots have expired.
• Maximum Count: Maximum number of snapshots to retain, even if the snapshots have not yet expired. If the number of snapshots in the repository exceeds this limit, the policy retains the most recent snapshots and deletes older snapshots.

After the configurations, Kibana shows us the API request version of these configurations. For this document, the request in code is like below;

PUT _slm/policy/daily-snapshot-backup-snapshot-training
{
  "name": "dsbst-{now}",
  "schedule": "0 30 4 * * ?",
  "repository": "test-backup-snapshot-training",
  "config": {
    "ignore_unavailable": true
  },
  "retention": {
    "expire_after": "30d",
    "min_count": 1,
    "max_count": 10
  }
}

And finally, the policy was created. On the configured time, Elasticsearch will be triggered to take a snapshot automatically.

Policy View

We may also start taking the snapshot policy manually anytime we want.

Snapshot API

• Get a List of Repositories

curl -XGET -u "username:password" "http://xxx.xx.xxx.xxx:9200/_snapshot?pretty"

• Get Repository Info

curl -XGET -u "username:password" "http://xxx.xx.xxx.xxx:9200/_snapshot/test_backup?pretty"

• Verify Repository

curl -XPOST -u "username:password" "http://xxx.xx.xxx.xxx:9200/_snapshot/test_backup/_verify?pretty"

• Get Snapshot Info

curl -XGET -u "username:password" "http://xxx.xx.xxx.xxx:9200/_snapshot/test_backup/kvs-snap-2020.12.07-cq65glqpsp6cd3xecrlfsq?pretty"

• Take Snapshot

curl -XPUT -u "username:password" 'http://xxx.xx.xxx.xxx:9200/_snapshot/test_backup/kvs-snap-2020.12.08?wait_for_completion=true{"indices": "backup_test*","ignore_unavailable": true}'

• Create Policy

curl -XPUT -H "content-type:application/json" -u "username:password" http://xxx.xx.xxx.xxx:9200/_slm/policy/test-backup-experimental -d '
{
"name": "<test-snap-{now/d}>",
"schedule": "0 0 4 ? * 6",
"repository": "test_backup",
"config": {
"indices":"backup_test*",
"ignore_unavailable": true,
"partial": true
},
"retention": {
"expire_after": "5d",
"min_count": 2,
"max_count": 4
}
}'

• Delete Snapshot

curl -XDELETE -u "username:password" "http://xxx.xx.xxx.xxx:9200/_snapshot/test_backup/test-snap-system-2020.12.14-gwgnmiycrcuiegvxsm4mga?pretty"
​

• Snapshot Status

curl -XGET -u "username:password" "http://xxx.xx.xxx.xxx:9200/_snapshot/test_backup/test-snap-system-2020.12.10-bd2dt2bkqi2dicd6dnb-8a/_status?pretty"

• Cat Snapshot

curl -XGET -u "username:password" "http://xxx.xx.xxx.xxx:9200/_cat/snapshots/test_backup?v"
​

• Restore a Snapshot

curl -X POST "http://xxx.xx.xxx.xxx:9200/_snapshot/test_backup/single-snap-az8kpc31sdouf5w4cuzq8a/_restore?pretty"

• Execute Policy (Take Snapshot with Policy)​

curl -XPUT -u "username:password" "http://xxx.xx.xxx.xxx:9200/_slm/policy/kvs_backup_with_system_indices/_execute"

That’s all. :) Thanks for reading! See you in my next post.