Deploy Amazon OpenSearch Service in a private VPC behind NGINX proxy and Cognito using CloudFormation

Introduction

Amazon OpenSearch Service is a managed service that makes it easy to deploy, operate, and scale OpenSearch clusters in the AWS Cloud. Amazon OpenSearch Service is the successor to Amazon Elasticsearch Service and supports OpenSearch and legacy Elasticsearch OSS (up to 7.10, the final open source version of the software).

OpenSearch is a fully open-source search and analytics engine for use cases such as log analytics, real-time application monitoring, and clickstream analysis.

When you create an OpenSearch domain via the AWS Console, it is recommended to create it inside a private VPC to avoid exposing it to the public internet. While this is a best practice in terms of security, it makes a bit tricky to access the OpenSearch Dashboards from outside the VPC.

AWS suggests three different options to resolve this problem:

• Use an SSH Tunnel
• Use an NGINX proxy deployed on an EC2 instance
• Use a Site-to-Site VPN

Although all three solutions are well documented by AWS (https://aws.amazon.com/premiumsupport/knowledge-center/opensearch-dashboards-vpc-cognito/), they could be really time consuming especially if you need to setup repeatedly an OpenSearch domain for testing purposes (keeping an OpenSearch domain running for long time can be quite expensive).

In this article we are going to describe how to implement the second solution using a CloudFormation template.

Architecture

The diagram below shows the architecture for the chosen solution and deployed using the CloudFormation template described in the next section:

The following components are deployed:

• A VPC
• A public subnet (where the NGINX proxy server resides) with an Internet Gateway
• A private subnet (where the OpenSearch domain will be placed)
• A Cognito User Pool with a corresponding Identity Pool
• An EC2 Instance where the NGINX proxy server is hosted

Implementation

The implementation is based on 1 main CloudFormation template (named Orchestrator) and 4 nested templates developed to deploy single components independently and to pass variables/outputs from one template to the other.

The nested templates are deployed in the following order:

• Infrastructure and Cognito (there are no dependencies between them)
• OpenSearch (it needs the VPC, the private subnet and the Cognito Pools)
• NGINX (it needs the VPC, the public subnet and the OpenSearch and Cognito urls)

The CloudFormation template to deploy the infrastructure is fairly simple and it is essentially responsible to deploy the resources listed below:

• A VPC
• A private subnet
• An Internet Gateway and a Route Table associated to the VPC (to expose the public subnet)
• A public subnet
• Different NACLs (to allow TCP traffic from/to the public subnet)

The CloudFormation template to deploy the Cognito resources needed to login into the OpenSearch Dashboards is a bit more complicated. It not only creates the User and Identity pools with corresponding roles for authenticated and unauthenticated users, but it also creates a default user with a temporary password using a Lambda function written in Python.

To achieve this, we have implemented a WiringFunction that is invoked (through a WiringFunctionInvocation) as soon as the Cognito pools and the Lambda execution role are created.

Once the Cognito pools are created, we are ready to setup and deploy an OpenSearch service domain. The code in this case is simpler and the template just creates the following resources:

• An IAM role that has the AmazonOpenSearchServiceCognitoAccess policy attached (this is a prerequisite to interact with Cognito)
• An OpenSearch domain
• A Security Group to allow communications over HTTPS

Now that we have our OpenSearch domain up and running, it’s time to setup and configure the NGINX proxy on the EC2 instance. In this case the template takes as input parameter not only the ids of the infrastructure resources needed for the EC2 instance, but also the CognitoUserPool and OpenSearch endpoint urls necessary to configure NGINX.

Two sections need to be noticed from the code snippet below regarding the creation of the EC2 instance:

• UserData
• CloudFormation::Init

The UserData field is used to install the NGINX package, generate the self-signed certificate and configure the init section. The CloudFormation::Init section instead, is mainly responsible for two tasks:

• Configure the default configuration file used by the proxy with the two endpoints received in input (to redirect the users either to Cognito login page or to the OpenSearch Dashboards)
• Ensure that the NGINX service is up and running

Once the main template has been successfully executed, you will be able to access the OpenSearch Dashboards using the urls provided as output by the template itself.