Accessing AWS ElastiCache (Redis) from different Amazon VPC(s) via AWS PrivateLink

Source: pixabay.com

We hit a roadblock during migration of an application (App-A) from AWS Account-A to AWS Account-B, because cross account AWS ElastiCache access isn’t straightforward and out of the box.

App-A is reading from ElastiCache managed in Account-A and after migrating App-A to Account-B accessing this ElastiCache isn’t possible. Why? That’s what this blog post is about.

The Problem

ElastiCache instance(s) are not accessible outside of the VPC in which they started in. Elasticache instance(s) are started under internal subnet(s) of the VPC meaning ElastiCache is only accessible by applications deployed in the same VPC as Elasticache.

Several approaches were considered to surmount this roadblock

• The first solution we considered was creating an IAM role with read permissions for ElastiCache in Account-A, creating an IAM role in Account-B which will assume the IAM role in Account-B. After migrating App-A to Account-B, App-A can assume the IAM role in Account-B and access ElastiCache (this solution was used but failed embarrassingly, due to lack of knowledge about ElastiCache being accessible only from within the VPC)
• Migrate ElastiCache as well from Account-A to Account-B. This solution was dropped, because in addition of migrating App-A and ElastiCache we also needed to migrate the application (App-B) writing to ElastiCache (migrating all three at the same time without downtime was challenging and would’ve skyrocketed the project timeline)
• Create another REST application (App-C) in Account-A which will access ElastiCache and serve data via HTTP REST endpoints(this application would have acted as a proxy, but it came at the cost of increased latency, extra maintenance and of course stretching project timeline)
• VPC peering (this was not possible because of VPCs in Account-A and Account-B had same CIDR range. Peered VPCs can’t have same CIDR range)
• Don’t migrate the application to other AWS account at all( would’ve been the last option)
• But with a bit more research and patience, finally, combination of Network Load Balancer (NLB) and AWS PrivateLink did the magic to solve the problem. I will only talk about this solution in particular as this was the one we used in the end.

Using Network Load Balancer(NLB) and AWS PrivateLink to enable cross account access of Elasticache.

First thing that needed to be confirmed was if Redis (ElastiCache) can even be accessed without using ElastiCache endpoints? The answer was Yes, Network Load Balancer (NLB) comes into play here, TCP traffic can be directed to the port of Target Group.

That’s what we did, we created an internet-facing NLB in Account-A targeting IP addresses of Redis ElastiCache instances also in Account-A. But this NLB can’t be internet-facing because the connection IS NOT protected/secure with authentication. This forced us to configure the NLB as an internal and we were back to square one (may be not to square one). Now the challenge was to connect to NLB in Account-A from Account-B.

One solution to the encountered problem was using NAT gateways for outgoing traffic from App-A in Account-B and whitelisting only those NAT gateways Elastic IP(s) from Account-B in the Security Group of internet-facing NLB in Account-A for inbound traffic. But this approach would have increased our costs significantly. We were looking for a cost-effective approach and AWS PrivateLink was the magic bullet we were looking for.

Steps:

1. Create an internal Network Load Balancer (internal-NLB) with the following configuration in Account-A:

• Load Balancer Protocol: TCP and Load Balancer Port: 6379

• Select relevant Availability Zones and internal subnets for the NLB

• Create a Target Group with Target Type: IP, Protocol: TCP and Port: 6379
• Select Health Check Protocol as TCP

• Register Targets: Specify IP address(es) of Redis ElastiCache node(s) as Target(s) for the previously created Target Group

2. Create an Endpoint Service and associate the above created NLB in Account-A

• Add Account-B to the whitelisted principals i.e add arn:aws:iam::<AWS-ACCOUNT-B-ID>:root

3. Create an Endpoint Interface in Account-B with following configuration:

• Service Category: Find service by name
• Enter the service name created in Account-A (The name can be found in Details section of Endpoint Services in Account-A)
• Select the relevant Availability Zones and internal subnets
• Create and attach a Security Group allowing inbound traffic from VPC’s CIDR range and outbound traffic to All.

4. Accept the Incoming Endpoint Connection to Endpoint Service in Account-A from Endpoint Interface created above in Account-B

5. Finally, use the global DNS name (usually the first one) from the Endpoint Interface in Account-B as the host address and port 6379 in App-A to access Redis ElastiCache.

This solution worked for us but increased the latency by ~400–500 microseconds which is acceptable for our use case.

There might be more possible solutions for this problem which I am not aware of and I would love to learn about them.