Elasticsearch snapshots backup/restore from s3 to another cluster.

Ezequiel Arielli
Dec 29, 2018 · 3 min read

Hello guys,

Today i will write about to take a elasticsearch snapshots in AWS cluster and restore this snapshot in another cluster.

Requirements:

  • s3 bucket to saved elasticsearch data
  • awscli
  • python27

First steps is create iam_role and policy to access the s3 bucket.

# create sts

vi es-sts.json{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": "es.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}

# create iam role
aws iam create-role — role-name es-snapshot-role — assume-role-policy-document file://es-sts.json

# create file policy

vi es-policy.json{
"Version":"2012-10-17",
"Statement":[
{
"Action":[
"s3:ListBucket"
],
"Effect":"Allow",
"Resource":[
"arn:aws:s3:::es-snapshots-rappi"
]
},
{
"Action":[
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject",
"iam:PassRole"
],
"Effect":"Allow",
"Resource":[
"arn:aws:s3:::es-snapshots-rappi/*"
]
}
]

# create iam policy with s3 access to saved elastic data.
aws iam create-policy — policy-name es-snapshot-policy — policy-document file://es-policy.json

# copy output arn in policy-arn
aws iam attach-role-policy — policy-arn “arn:aws:iam::581653423581:policy/es-snapshot-policy” — role-name es-snapshot-role

# vi es-snapshot.py

#### if you es endpoint is only http you configure is_secure=False in ESConnection

BACKUP

# Run python — This python register you s3 bucket mapping in _snapshot path.

python es-snapshot.py

Output:

{"acknowledged":true}

# Generate snapshot

curl -XPUT 'https://vpc-test-fgo4ad36abibtmilnadvisjdfiskd.us-west-2.es.amazonaws.com/_snapshot/es-backups/mysnapshot'

Output:

{"accepted":true}

# Check files saved in s3 bucket

[root@ip-19-21-30-77 ~]# aws s3 ls s3://es-snapshots-rappi
PRE indices/
2018-12-27 21:59:21 172 index-0
2018-12-27 21:59:21 8 index.latest
2018-12-27 21:59:21 444 meta-wKOsJBfRT3qOac3bI_R_qg.dat
2018-12-27 21:59:21 222 snap-wKOsJBfRT3qOac3bI_R_qg.dat

# Check the state about snapshots in the domain.

curl -XGET 'https://vpc-test-fgo4ad36abibtmilnadvisjdfiskd.us-west-2.es.amazonaws.com/_snapshot/es-backups/_all?pretty'

# Look the all repositorys with snapshots.

curl -XGET 'https://vpc-test-fgo4ad36abibtmilnadvri56su.us-west-2.es.amazonaws.com/_snapshot?pretty'

IMPORTANT!!!!!!!

Registering a snapshot directory is a one-time operation, but to migrate from one domain to another, you must register the same snapshot repository in the old and new domain.

Restore

git clone https://github.com/nightmareze1/efk_stack.git

# Register new efk_cluster in s3 bucket

[root@ip-10-91-41-215 elk]# curl -XPUT 'http://localhost:9200/_snapshot/es-backups/' -H 'content-type: application/json' -H "Accept: application/json" -d @s3.json

IMPORTANT!!!!!!!! :

If you need restore the all indices include .kibana you need delete the all indices, because don’t work kibana is the index of the system and is running.

# RESTORE ALL INDICES

# Restore — Single indice

If you have the indices in state yellow, it’s possible you need configure the replicas again.

# You need reconfigure the replicas in the indice.

Results:

Troubleshoot, if you have problems in restore process you can check information with this commands:

Finish add the indices in kibana and check correct status.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store