Memorystore Redis access through VPC peering

A GCP user wanted to create a Memorystore Redis attached to a VPC. This was easy enough. The user then wanted to create another VPC in a separate project and have VMs in that VPC access the original Memorystore Redis. Can this be achieved and if so how?

This article walks us through a journey where we set up such an environment and got it working. The story takes us through the working to get there and explains some of the puzzles involved.

We will start with a simple diagram of our base peered network:

Let’s call this the baseline. It does not represent our final desired outcome, but let us see if we can achieve this environment.

• Create a GCP project called project-i-1.
• Delete the VPC network called default.
• Create a VPC network called vpc-1.
• Create a subnet in vpc-1 called subnet-1 with IP address 10.1.0.0/16 in region us-central1.
• Create a VM called vm-1 in project-i-1 attached to the subnet called subnet-1 belonging to vpc-1. In my example, we found that this was given IP 10.1.0.2.
• Create a GCP project called project-i-2.
• Delete the VPC network called default.
• Create a VPC network called vpc-2.
• Create a subnet in vpc-2 called subnet-2 with IP address range of 10.2.0.0/16 in region us-central1.
• Create a VM called vm-2 in project-i-2 attached to the subnet called subnet-2 belonging to vpc-2. In my example, we found that this was given IP 10.2.0.2.
• Create a VPC peering definition in project-i-1 called vpc-1-to-vpc-2 that maps vpc-1 to vpc-2.

• Create a VPC peering definition in project-i-2 called vpc-2-to-vpc-1 that maps vpc-2 to vpc-1.

• Check that the status of the VPC peerings in both projects are “Active”.

• Create a firewall rule in vpc-1 allowing incoming SSH port 22 from 0.0.0.0/0.

• Create a firewall rule in vpc-2 allowing incoming all ports/protocols from 10.0.0.0/8. This includes subnet-1.

At this point we can SSH into vm-1 and execute a ping against vm-2 (10.2.0.2) and it will succeed.

This illustrates that we have a network path from vm-1 to vm-2 which crosses projects and VPCs. Now it is time to extend our story and add Memorystore Redis.

• Enable Memorystore for Redis in project project-i-2.
• Create an instance of Memorystore for Redis. This created an instance at 10.34.175.91. This will likely be different for you. Make a note of the IP for future use.

• Create a firewall rule in vpc-2 allowing incoming SSH (port 22) from 0.0.0.0/0.
• SSH into vm-2.
• Install the Redis Command Line Interface (CLI).

sudo apt-get install redis-tools

• Run the Redis cli

export SERVER=[REDIS-1 IP — 10.34.175.91]
redis-cli -h $SERVER

• Save a name/value pair.

set mykey myvalue

as we see, this works just fine.

• SSH into vm-1.
• Install the Redis Command Line Interface (CLI).

sudo apt-get install redis-tools

• Run the Redis cli

export SERVER=[REDIS-1 IP — 10.34.175.91]
redis-cli -h $SERVER

It fails. We appear to have no path to 10.34.175.91 from a VM attached to vpc-1.

Let’s see if we can’t explain this story. Reviewing, we see that vm-2 can connect to Redis but vm-1 can’t. Let’s first ask why we think that vm-1 might be able to connect to Redis? We have a gut feel that it might work because vpc-1 and vpc-2 are peered. What this tells us is that things attached to vpc-1 have a network path to things connected to vpc-2. We have demonstrated that as we can ping vm-2 from vm-1. When we created the Redis instance, we named vpc-2 as the VPC from which requests to Redis would be allowed. If we look at the output from a description of the Redis server, we see an entry called authorizedNetwork which names vpc-2.

This might cause us to think that Redis is attached to vpc-2 and also that vpc-1 and vpc-2 are peered and hence a VM attached to vpc-1 should be able to reach Redis. Unfortunately, the story is not quite that simple. If we dig deeper, we will find that Redis is actually attached to its own private and un-named VPC and that when we create the Redis instance, it actually creates a VPC peering between the Redis un-named VPC and the VPC to which Redis is being attached (vpc-2 in this case).

A more appropriate diagram would thus look like:

This fully explains why vm-2 is able to access Redis. The VPC called vpc-2 is peered with the un-named VPC that Redis is using. Since vm-2 is attached to vpc-2 it has a peered connection to the Redis VPC and hence to Redis itself.

And we have one final concept … VPC peering is not transitive. Even though vpc-2 is peered with the Redis VPC and vpc-1 is peered with vpc-2, that does not mean that vpc-1 is thus peered with the Redis VPC. This lack of transitiveness is documented here. There we will find the following:

Only directly peered networks can communicate. Transitive peering is not supported. In other words, if VPC network N1 is peered with N2 and N3, but N2 and N3 are not directly connected, VPC network N2 cannot communicate with VPC network N3 over VPC Network Peering.

This is one plausible reason that we can’t connect from vm-1 to Redis. Does this mean that we simply can’t connect from vm-1 to Redis? Yes and no. We can’t connect directly but we can connect through a simple proxy. Our high level story will be to create a low-end Compute Engine in the vpc-2 VPC. Since this Compute Engine is attached to the vpc-2 VPC it will have direct access to the Redis server. On this Compute Engine we will install the open source package called nutcracker. Nutcracker is a proxy that is cognizant of Redis. It will receive incoming Redis requests and efficiently forward those to the Redis server. Since the proxy Compute Engine is just a Compute Engine attached to vpc-2 it will be available to any other VPCs that are peered to vpc-2 which, in our story, will include vpc-1.

• Create a VM called redis-proxy in project-i-2 attached to the subnet called subnet-2 belonging to vpc-2. In my example, we found that this was given IP 10.2.0.3.
• SSH into redis-proxy.
• Install nutcracker.

sudo apt-get update
sudo apt-get install nutcracker

• Create the nutcracker configuration file called nutcracker.yaml.

redis-1:
 listen: 0.0.0.0:6379
 redis: true
 servers:
 - [REDIS-1 IP]:6379:1

• Start the nutcracker application.

sudo nutcracker --conf-file nutcracker.yaml

• On vm-1, run a Redis command pointing to redis-proxy.

export SERVER=10.2.0.3
redis-cli -h $SERVER
get mykey

This test succeeds and brings us to the end of the story. We have now achieved interaction with a Memorystore Redis instance that is defined to a VPC that is peered to another VPC which hosts the Redis client application.

There are other possibilities that we might consider to achieve our functional goal, core among these is the idea of using Shared VPC and creating our Memorystore Redis in a host project and making it available through service projects.

As a companion to this article, here is a video illustrating a walk-through of the story described here: