CDK. Secure Access AWS VPC-based Elasticsearch Cluster Locally without SSH Keys
So, your Elasticsearch cluster is in VPC…
This means that to get access to it, you should be “somewhere around”. And it can be a challenge when you have a large number of people (more than 2) who want to have access to it.
When we started our project, just two of us needed access to the Elasticsearch (ES) cluster. And due to security concerns from our client-side, we placed cluster in VPC.
We used the standard approach with bastion host in the public subnet and created a tunnel to cluster via it. We used direct connection fairy rarely, as we need it only when something went wrong. All other times using debug information in our lambda response was more than enough. It was easy to share keys between the two of us. Then QA has joined our team, so we were 3 of us. We have put keys into confluence. At the beginning of this summer, our team has grown twice, so it became too complicated to deal with keys, plus as people are working from their homes, managing Security groups has become more difficult. Plus, AWS itself recommends not use the bastion host and use the SSM Session manager.
At that moment, search over the internet did not provide any out-of-box solution for injecting to the CDK template, so we started to build our own bicycle…
Original Solution
Our original solution with an instance in public subnet looked like this:
New Solution
Your new solution should look like the one in the picture below.
I don’t want to call it architecture, as I’m not sure how it is working until the end.
Some sources say that the Session manager utilized SSH connection, but I could not find it in the official documentation. Please feel free to point me towards any information in the comments 🙏.
Benefits what we were expecting:
1. No more managing of SSH keys.
2. No public IP for bastion host.
3. Access from anywhere.
Here are the steps that we have on our bastion instance:
- Created an EC2 role to be able to connect via Session manager and assign it.
2. Update image to Amazon Linux v2.
3. Remove keypair. (it was somewhere there…)
4. Change the subnet to a private one.
5. Move instance from public SG to the same SG as ES cluster.
6. Create haproxy.config content. We will set up redirection of traffic coming to the bastion host to our elastic search vpc cluster in its configuration.
7. Added init configuration to an instance. In which we will create a haproxy.config file (line 7–10) and enable haproxy (line 13–17).
8. Increase init timeout to 15 minutes. Otherwise, in our case, CF was not receiving a success signal and was reverting deployment.
9. Overwrite logical id of instance. Otherwise, user data will fail to get init metadata for execution.
10. Create and assign user data pathing stack name, region, and new logical id for our instance.
In the end, our complete construct should look like this:
And when you did everything, we just added this construct into our current stack and deployed it.
To connect via Session manager, we have created two simple scripts for our Mac and Windows users.
Your terminal will look like this during connection:
And voila…
…you are in Kibana and able to execute requests!
To be able to execute this script, you should do the next setup:
- Install AWS CLI v2+ (here).
- Install Session Manager plugin for the AWS CLI (here).
- Set up AWS SSO for accounts where you want to use it (here).
- Execute the script, passing your profile name as a parameter.
Be sure that your role has policies in place to be able to execute it. Please limit it to specific resources before applying to your account:
Now can connect to http://localhost:9200. Unfortunately, connection over HTTPS is causing a cert issue. But we have promised ourselves to fix it later.
As a result:
- No more SSH keys.
- No more SG updates with new IP addresses.
- No more Public IPs.
- Only temporary credentials for getting access to ES.
- HTTPS not working yet…
We are ACTUM Digital and this piece was written by Oleksii Ivanchenko, Lead Cloud Architect of Apollo Division. Feel free to get in touch.
