How to Create a Manual Snapshot Repository in OpenSearch on AWS

This tutorial focuses on configuring the OpenSearch service running on AWS, not on standalone or on-premises installations.

While OpenSearch on AWS provides automatic snapshots that keep 14 days of daily snapshots in AWS S3 buckets, these are managed automatically and cannot be manipulated.

To take manual snapshots, you need to create a snapshot repository first. Here’s a detailed guide on how to define a new repository.

1. Create Your S3 Bucket

- Create a new S3 bucket and name it “open-search-backups”.

2. Create a New IAM User to Sign API Requests

- Create a new IAM user for sending API requests to OpenSearch.
- This user will be used to send the request for adding a new repository as an API call. Let’s name this user `open-search-admin`.

arn:aws:iam::12345678:user/open-search-admin

- Create the Access Key and Secret Key credentials for this user and use them in your API request (details to follow).
- Attach two policies to this user: one for `PassRole` and another for `EsHttpPut`. Create and attach the following policy:

{
 "Version": "2012–10–17",
   "Statement": [
     {
       "Effect": "Allow",
       "Action": "iam:PassRole",
       "Resource": "arn:aws:iam::12345678:role/OpenSearchSnapshotRole"
     },
     {
       "Effect": "Allow",
       "Action": "es:ESHttpPut",
       "Resource": "arn:aws:es:eu-central-1:12345678:domain/my-open-search-domain"
     }
   ]
 }

3. Create a New IAM Role and Policy

- Create a new role for OpenSearch to access S3. This role will be used in the API request and will allow OpenSearch to access and create the repository. Let’s name this role `OpenSearchSnapshotRole`.

arn:aws:iam::12345678:role/OpenSearchSnapshotRole

- Attach a policy to this role with the necessary permissions to access S3. Here’s the policy JSON:

{
 "Version": "2012–10–17",
 "Statement": [
     {
     "Effect": "Allow",
     "Action": [
     "s3:PutObject",
     "s3:GetObject",
     "s3:DeleteObject",
     "s3:ListBucket"
     ],
       "Resource": [
       "arn:aws:s3:::open-search-backups",
       "arn:aws:s3:::open-search-backups/*"
       ]
     }
   ]
 }

4. Map the OpenSearch Role with the AWS User

• Map the IAM role within OpenSearch RBAC (Role-Based Access Control).
  - Navigate to OpenSearch > Security > Roles > manage-snapshot.
  - Find the “map users” action and enter your IAM user’s ARN into the mapping field.

- If this mapping is not done, you will encounter the following error:

{
 "error": {
   "root_cause": [
     {
       "type": "security_exception",
       "reason": "no permissions for [cluster:admin/repository/put] and User [name=arn:aws:iam::12345678:user/open-search-admin, backend_roles=[], requestedTenant=null]"
       }
     ],
       "type": "security_exception",
       "reason": "no permissions for [cluster:admin/repository/put] and User [name=arn:aws:iam::12345687:user/open-search-admin, backend_roles=[], requestedTenant=null]"
       },
 "status": 403
 }

5. Run Your API Request

- With all components set up, you can now add the new repository to S3 by making a PUT API call.
- Using Postman is recommended due to its support for AWS Signature authentication. Set the credentials for the user `open-search-admin` and configure the authorization as shown below:

- Ensure the service name is set to `es` (for Elasticsearch) and run the command.

curl - location - request PUT 'https://my-open-search-domain.eu-central-1.es.amazonaws.com/_snapshot/aws-s3' \
 - header 'Content-Type: application/json' \
 - header 'X-Amz-Content-Sha256: 123456789123456789123456789' \
 - header 'X-Amz-Date: 20240525T114900Z' \
 - header 'Authorization: AWS4-HMAC-SHA256 Credential=BAC123ABC123BAC123ABC123/20240525/eu-central-1/es/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-content-sha256;x-amz-date, Signature=123123123123123123123123132' \
 - data '{
 "type": "s3",
 "settings": {
 "bucket": "open-search-backups",
 "base_path": "a",
 "region": "eu-central-1",
 "role_arn": "arn:aws:iam::12345678:role/OpenSearchSnapshotRole"
 }
 }'

- If everything is configured correctly, you should receive the following response:

{
 "acknowledged": true
 }

- You can now see your newly created snapshot repository under OpenSearch.

With this setup, you can take new manual snapshots in OpenSearch and store them in your specified S3 bucket.

If you need, you can delete a mistakenly create repository like below:

curl --location --request DELETE 'https://open-search-domain.eu-central-1.es.amazonaws.com/_snapshot/failed-name-aws-s3' \
--header 'Content-Type: application/json' \
--header 'X-Amz-Date: 20240525T131714Z' \
--header 'Authorization: AWS4-HMAC-SHA256 Credential=ABC123ABC123ABC123/20240525/eu-central-1/es/aws4_request, SignedHeaders=host;x-amz-date, Signature=123123123123123123123123123123' \
--data ''