Everything You Always Wanted to Know About VPC Peering* (*But Were Afraid to Ask)

TL:DR

The article provides an overview of Google Cloud VPC network peerings, their anatomy, major misconceptions, and some watchpoints, so that users can learn how to use them wisely, while designing their infrastructures.

Context

Virtual Private Cloud (VPC) peerings allow to connect multiple VPC networks at high speed, low latency and no extra costs.

Given how powerful they are and how easy they can be managed, users often decide to adopt them as the standard mechanism to connect their networks in GCP.

While working with many customers, I’ve always noticed that many concepts around VPC peerings are often misunderstood and can become a source of confusion. This unfortunately translates into bad design decisions, which need to be often fixed when the project has already started.

The goal of this article is to make clarity on these misconceptions, hopefully helping users designing their networks more wisely and consciously.

Let’s explore together some characteristics of VPC peerings.

Administrative Domains

Users get often confused when they see VPC peerings represented with single lines in diagrams, neglecting the fact that their administration is instead spread across different administrative domains.

In order for VPC peerings to be established, each VPC administrator needs to explicitly authorize their network to connect to the other. Whether the administrators can be the same or different, two separate configurations need to happen in GCP, before the VPC peering comes up.

In this example, the VPCs hosted two different projects, owner by two different administrators. They both need to create a peering to the other VPC, before the peering is formed.

This is what happens in GCP if administrators only authorize the peering from one network to the other:

The peering is still inactive, since only one administrative domain has authorize the peering.

While this is what a healthy peering looks like, once both networks have authorized the communication.

Peerings are both “Active”, as both administrators have authorized the VPC peering.

Peerings are non-transitive

Let’s make another example: let’s say we have three VPCs: A, B and C.
A and B peer with one another, and B and C peer with one another.

While hosts in VPCs directly peering will communicate, this won’t be true for hosts connected to A and C.

The three VPCs are peering. Machine A can communicate with B, and B with C. Anyway A cannot communicate with C, without additional, VPC peerings between A and C.

We see the use-case happening quite often with:

• Hub-and-spoke topologies: it’s not possible to let different spokes communicate through the hub, unless VPC peerings between the spokes get created (or a network appliance in the hub mediates the communication between the spokes)
• Managed Google Services: many Google managed services leverage peerings to connect Google managed VPCs (where the manage service runs) to the users’ VPCs. Need examples? GKE private clusters, Cloud SQL, NetApp CVS Volumes are just the most popular out of many others. What if users VPCs also peer with other VPCs, from which hosts need to access the managed services?

“Alright, what’s the problem? Let’s add more peerings!” …not really!

Here are some important considerations:

• Google managed VPCs can’t be configured by users, so you cannot create -for example- extra peerings to other VPCs.
• Most importantly, the number of VPC peerings is limited! We’ll talk more about this in the next paragraphs.

It’s important to understand that with limitations come also alternatives, which can help us to either avoid or get around possible issues. These are the ones I propose more often:

• Try to reduce the number of VPCs (so of VPC peerings!).
• Evaluate the substitution of peerings with managed HA VPNs between VPCs.
• Add network appliances or proxies, in order to break and re-initiate the connection, so we can jump through multiple peerings at ones.
• Consider leveraging Private Service Connect (PSC) for Internal Load Balancers (ILBs), in order to publish private endpoints for your services, which can be consumed from other VPCs, without any need for additional VPC network peerings.

There’s really not a good or bad answer here. Work with your good sense to build the solution that fits your needs.

Not everything travels through peerings

It’s import to note that not all information are carried over peerings. Among others, the most relevant are:

• GCE internal DNS names, which cannot be referenced by a peering VPC.
• Firewall Tags and Service Accounts, which cannot be referenced from a firewall rule created in another peering VPC.
• Cloud DNS zones and records: resolution of records defined in Cloud DNS doesn’t happen through peerings, but through Google API calls. To allow cross-VPC Cloud DNS hostnames resolution use DNS peering instead.
• Routes whose next hops are default Internet gateway.
• Custom static routes using network tags.
• Firewall rules are never exchanged via peering.
• Ingress firewall in one VPC network cannot identify sources by source tag or source service account from the other VPC network
• Private Service Connect (PSC) endpoints, which cannot be consumed by other peering VPCs (…does it even makes sense?)

Importing and Exporting Routes

Peered VPCs always automatically exchange their private IP address subnet routes.

Additionally, you can configure peered VPCs to exchange other routes:

• Exchange routes related to subnets containing Privately Used Public IPs.
• Exchange custom routes, for example, custom static routes, or custom dynamic routes learned via BGP from Cloud VPN or Cloud Interconnect.

In both cases, the peering configuration provides a dedicated option, which allows to export and/or import routes from and to another VPC.

The following is an example, taken from the GCP Console:

For both networks, peerings have been set to import and export, both their custom routes and their subnets including PUPI addresses.

The configuration may vary, depending on the user’s needs. At the very minimum, one VPC peering needs to allow the export of the routes, while the other needs to authorize their import.

Let’s see a graphical example, related to dynamic routes:

Through peering settings, dynamic (custom) routes are exported from VPC A to VPC B, and imported from VPC B to VPC A

The administrator of VPC A wants to share the custom (dynamic) routes learned from Cloud VPN (through Cloud Router) with VPC B. To do so, the administrator of VPA A exports the custom routes, editing its peering (from A to B). To complete this process, the administrator of VPC B imports the custom routes (from B to A).

Peering groups limits and scalability

This is the most common misunderstanding, but it had to be presented after covering fundamental VPC peering concepts first.

Let’s start getting familiar with a new term: a Peering Group (PG) is the set of VPC networks connected through direct VPC peerings.

It’s important to note that

• This concept (and the related cardinality of the elements) is specific to each VPC: in other words, each VPC calculates its own usage for each peering group limit.
• We’re talking about direct peerings. Given that peerings are not transitive, what counts are the networks directly connected to the VPC we’re taking in consideration.

Let’s see an example:

Different Peering Groups and their cardinality for the same set of interconnected VPCs.

The image shows different VPCs connected through VPC peerings. For each VPC we see a specific peering group, represented with a different color, and an associated cardinality.

There are 5 VPCs in the diagram shown above, so 5 Peering Groups.

So, why Peering Groups are so important to us?

Because the number of different GCP elements within a peering group is limited.

This includes:

• VPC peerings (at the moment of writing, it is 25)
• GCE instances
• Subnets and alias ranges
• ILBs forwarding rules
• Static and dynamic routes

The algorithm used to calculate these limits is publicly available in the official Google Cloud documentation.

While many of these numbers may seem quite high, they happen to be hit soon if the network is not designed properly and VPC peerings are not used wisely.

These are soft limits and can be exceptionally increased through a request to Google Sales and Cloud Support. It’s important to keep in mind that increases are not guaranteed and, often, rising a number will likely cause others to be lowered, given how these objects map together in the underlying Google SDN infrastructure.

Final Remarks

VPC peerings are very useful and powerful, but users should use them wisely, in order to avoid odd surprises. When you design your infrastructure:

• Try to minimize the number of VPC peerings
• Foresee your future needs. Do you need to connect to managed services? Do you need to allow direct communication between your spokes? Peerings may not necessarily be the right solution for you. Evaluate using PSC, proxies or managed VPNs.
• Focus on your requirements and on mechanisms to make your network scale better; less on limits. Limits are a continuously moving target and they can be exceptionally increased upon request, if there’s a good reason for it. You shouldn’t design your network, basing your decisions (primarily) on limits. Start from your requirements and work to make your network design more scalable. If you did a good job, you almost certainly won’t hit Google’s network limits.
• Keep numbers under control. While working through your design, remember to keep in consideration multiple dimensions for each Peering Group: this includes VPC peerings (also counting the ones that Google manages), the number of VMs, the forwarding rules, and much more!
• Don’t take numbers in this article as a given. Periodically, check the official documentation and engage with Sales and Google Cloud Support to learn more about recent updates. Numbers continuously evolve with the infrastructure.

While this information is included in the official documentation, it took me quite some time to bring different pieces together and gain enough experience to write this. Hopefully, this article is useful so you don’t make the same mistakes we often see!

Thank you

Simone Ruffilli

and

Gerritd

for the support and insightful reviews.