Elasticsearch snapshots backup/restore from s3 to another cluster.

Ezequiel Arielli

3 min readDec 29, 2018

Hello guys,

Today i will write about to take a elasticsearch snapshots in AWS cluster and restore this snapshot in another cluster.

Requirements:

s3 bucket to saved elasticsearch data
awscli
python27

First steps is create iam_role and policy to access the s3 bucket.

# create sts

vi es-sts.json{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "es.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

# create iam role
aws iam create-role — role-name es-snapshot-role — assume-role-policy-document file://es-sts.json

# create file policy

vi es-policy.json{
    "Version":"2012-10-17",
    "Statement":[
        {
            "Action":[
                "s3:ListBucket"
            ],
            "Effect":"Allow",
            "Resource":[
                "arn:aws:s3:::es-snapshots-rappi"
            ]
        },
        {
            "Action":[
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "iam:PassRole"
            ],
            "Effect":"Allow",
            "Resource":[
                "arn:aws:s3:::es-snapshots-rappi/*"
            ]
        }
    ]

# create iam policy with s3 access to saved elastic data.
aws iam create-policy — policy-name es-snapshot-policy — policy-document file://es-policy.json

# copy output arn in policy-arn
aws iam attach-role-policy — policy-arn “arn:aws:iam::581653423581:policy/es-snapshot-policy” — role-name es-snapshot-role

# vi es-snapshot.py

#### if you es endpoint is only http you configure is_secure=False in ESConnection

BACKUP

# Run python — This python register you s3 bucket mapping in _snapshot path.

python es-snapshot.py

Output:

{"acknowledged":true}

# Generate snapshot

curl -XPUT 'https://vpc-test-fgo4ad36abibtmilnadvisjdfiskd.us-west-2.es.amazonaws.com/_snapshot/es-backups/mysnapshot'

Output:

{"accepted":true}

# Check files saved in s3 bucket

[root@ip-19-21-30-77 ~]# aws s3 ls s3://es-snapshots-rappi
PRE indices/
2018-12-27 21:59:21 172 index-0
2018-12-27 21:59:21 8 index.latest
2018-12-27 21:59:21 444 meta-wKOsJBfRT3qOac3bI_R_qg.dat
2018-12-27 21:59:21 222 snap-wKOsJBfRT3qOac3bI_R_qg.dat

# Check the state about snapshots in the domain.

curl -XGET 'https://vpc-test-fgo4ad36abibtmilnadvisjdfiskd.us-west-2.es.amazonaws.com/_snapshot/es-backups/_all?pretty'

# Look the all repositorys with snapshots.

curl -XGET 'https://vpc-test-fgo4ad36abibtmilnadvri56su.us-west-2.es.amazonaws.com/_snapshot?pretty'

IMPORTANT!!!!!!!

Registering a snapshot directory is a one-time operation, but to migrate from one domain to another, you must register the same snapshot repository in the old and new domain.

Restore

git clone https://github.com/nightmareze1/efk_stack.git

# Register new efk_cluster in s3 bucket

[root@ip-10-91-41-215 elk]# curl -XPUT 'http://localhost:9200/_snapshot/es-backups/' -H 'content-type: application/json' -H "Accept: application/json" -d @s3.json

IMPORTANT!!!!!!!! :

If you need restore the all indices include .kibana you need delete the all indices, because don’t work kibana is the index of the system and is running.

# RESTORE ALL INDICES

# Restore — Single indice

If you have the indices in state yellow, it’s possible you need configure the replicas again.

# You need reconfigure the replicas in the indice.

Results:

Troubleshoot, if you have problems in restore process you can check information with this commands:

Finish add the indices in kibana and check correct status.