Neiman Marcus Tech
Published in

Neiman Marcus Tech

Building a Secure AWS managed ElasticSearch cluster using Terraform

Over the last few months I’ve had the opportunity to deploy an ElasticSearch cluster using AWS ES service at Neiman Marcus. Currently Neiman Marcus’s ES cluster is managed Ec2s and it was time to move towards a managed service. After careful consideration we chose to go with AWS’s offering (instead of elasticsearch.io), mainly because AWS ES offers almost everything we need for our use-case, is much cheaper and we’d prefer having our cloud billing under one roof. There are a lot of articles out there comparing the pros & cons of AWS ES so in this post I’d like to focus on the How instead of the Why.

AWS ES offers multiple levels of security which, depending on your usecase can help you lockdown the cluster and protect against threats. ES can be setup to be accessible over the public internet but we chose to lock it down using VPC based ES cluster. This gives us better control over the access via Security Groups. There are a few downsides to this approach which you can read about here https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-vpc.html

Pre-requisites

  1. AWS account with VPC
  2. Terraform v0.12 (you can use older version, syntax might change)

Basic resources you would be creating

  1. ElasticSearch domain
  2. Cognito User pool and Identity pool for Kibana (optional)
  3. IAM roles

A) ElasticSearch Domain

Here is the snippet of the aws_elasticsearch_domain resource.

B) Setting up Cognito User pool and Identity pool for Kibana

Note: The conditional ‘count’ logic is something we added for our usecase and is totally optional.

Some key points to remember

  1. Once AWS ES has the necessary privileges, it will create an App Client ID in the user pool you submitted and link it to the identity pool you submitted. You will see it under Identity pool > Authentication Providers

2. Accessing Kibana could be an issue if you are using VPC based ES cluster. Since this type of ES cluster isn’t accessible over the internet you need to be running it from within the VPC. There are 2 ways of doing it…

a) Connect to it over a VPN: This is the best way to connect if you already have a VPN that has access to the VPC

b) Connect via SSH tunneling: For this workaround you’ll need a bastion host that is part of the ES cluster security group and a pem key to the bastion host.

1. Create a file ~/.ssh/config
2. run ssh -N estunnel and leave it running…3. Go to https://localhost:9200/_plugin/kibana to access Kibana

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store