GCP Routing: Delving into the Labyrinth
Do peering subnet routes have precedence over VPN HA routes? Can I override local subnet routes somehow? When does routes priority really matter? If you ever questioned yourself with some of the above, then you might find this article interesting…. and nope, I won’t spoiler the answers in this abstract 😊
On Google Cloud, routing your traffic correctly and efficiently hinges on understanding the intricate dance of route types, locality and priority. When traversing the complex landscape of enterprise network topologies, where peerings, VPNs and interconnects are in place, deciphering which route reigns supreme for a specific target destination becomes paramount. In this article we’ll venture deep into the GCP routing logic, in order to understand the exact route precedence ordering, unveiling the interplay of CIDR prefixes, route priority, locality and so on.
The Contenders: A Hierarchy of Routes
In the Google Cloud we can identify various classes of routes. Let’s have a quick look at them.
- Special routing paths: These are special routes that cannot be ever overridden and are managed directly by Google Cloud routing layer. Such routes are neither visible within the management console nor via gcloud or APIs. Nevertheless, they exist and play crucial roles in various scenarios. In our context, you can think of them as invincible wizards that play magic spells affecting specific routing conditions.
- Subnet routes: these are routes created automatically by the Google Cloud Networking layer, every time a subnet is added to a VPC. Each subnet has a local route that affects the VPC they lay in and any peered VPC. Think of them as very glorious knights of the kingdom, each one is defending his own subnet within his VPC.
- Static routes: these are manually added routes by the network engineers. We can distinguis between Policy Based Routes (PBR) and Classic Static Routes. The former are very powerful and take precedence over the latter, no matter what. Such routes have the ability to change the traffic flow in specific circumstances. You can see them as kings (PBR static routes) and kingdom’s warriors (classic static routes).
- Peered routes: when two networks VPC-A and VPC-B are peered together, they exchange routes. VPC-A will receive all the subnet routes belonging to VPC-B and vice versa. We can distinguish between peering subnet routes and peering static and dynamic routes. The former are assimilable to glorious knights of the allied kingdom, while the latter are more similar to allied warriors.
- BGP routes: these dynamic routes originate from Cloud Routers, exchanging routing information with on-premises networks or other VPCs via BGP. They can also be acquired via peering. In our scenario, those are warriors traveling distinct realms.
The Dueling Deities: Precedence Rules the Realm
Now that we have listed the various route types, we start looking at how Google Cloud will handle routing.
When a packet wants to reach a specific destination, (i.e. a traveller is headed to a specific kingdom), the route it’ll take is ruled by the following criteria:
- Route Type Precedence
- Route specificity
- Route next-hop’s locality
- Route priority
- Tie breakers
This means that the GCP routing system will first check if a routing decision can be made by only looking at the route-type precedence order. If not, it’ll look at the route specificity. If still there is no clear route to be taken, it’ll consider the next-hop locality. Again, if no clear decision can be made (i.e. multiple contending routes are available), it’ll look at the route priority, and so on. Each evaluation step has its own logic, that we’ll describe below. Let’s go!
First Round: Route type precedence
The first routes that are taken into account are the Special Return Paths, no exception. If the packet matches any of the target destinations within the special return paths, the packet will be routed via the internal GCP routing system and the traveller is teleported via the wizard’s magic to its destination. For us as Network Architect, that’s really magic as there is literally nothing we can do to override such routes.
If no special return path is relevant for the destination IP, then the packet is processed via the Policy Based Routes (PBR) affecting the VPC the packet is traversing, if any. In this case, the packet is evaluated against all the PBR rules, following the PBR priority values (PBR are associated with a unique priority value and there could not be multiple PBR with the same priority). When a packet matches a PBR rule criteria (i.e. its source IP, destination IP and protocol do comply with the PBR values) the packet is handled via the specified next-hop. As per today, there could be only a possible next-hops for a PBR: an internal load balancer (forwarding rule) within the same VPC where the PBR lives. However, a PBR can also be configured to make the packet skip all the other PBRs and continue the routing via next routing options.
Then GCP evaluates the Subnet Routes (belonging to the local traversing VPC). If a packet is heading to an IP-address which belongs to one of the local subnets, the packet is delivered to the local subnet. In other words, if there is a glorious knight defending the village (subnet) where the traveller is headed to, then the traveller is escorted to its destination by the knight.
Eventually, GCP looks at the Peered Subnet Routes (i.e. villages of allied realms). If a packet needs to reach a peered network subnet, it’ll be processed via the peered subnet route right away. Please note that this last step does not apply to dynamic routes exchanged via peering: these are not prioritized and will “fight” against the other possible routes in the next round.
If no applicable route belongs to any of the route types above, the routing decision is not taken at this stage. This occurs when the only applicable routes belong to on of the remaining classes: static routes (non PBRs), BGP routes (acquired via Interconnect, VPNs) and dynamic peered routes. This means that there will be a battle among kingdoms’ warriors (classic static routes), traveling warriors (dynamic routes, either exchanged via BGP or via peering) to determine which one will prevail. The fight will take into account their combat specialty (specificity), the knowledge of the combat field (locality) and their experience (priority).
Second round: Route specificity
When a packet is headed to a specific IP address and there are multiple routes contending that IP, Google Cloud will prefer the route that is most specific. In other words, the longest CIDR prefix takes precedence (as it does in classic routing). For instance, a route for /16 is less powerful of a a /24 route if both encompass the destination IP of the packet. Please note that advertising a specific route does not necessarily mean the path to the destination will be shorter than a more broader advertisement. Routing experts know that very well.
In our analogy, we can look at it as multiple warriors (routes) contending to escort the traveller (packet) to their destination. In this case, the warrior fighting in their own specialty (i.e. most specific route) will prevail against the others.
At this stage, most of the packet flows should already be handled by either route type or CIDR specificity rules. However, there still are cases where further analysis is needed. This happens when we deal with more complex network topologies involving VPNs, peerings, interconnect and static routes, playing together. For instance, consider the following topology:
In Figure 2, VPC-A is peered with VPC-B (with routes export/import active), and a HA VPN (with BGP) exists between VPC-B and VPC-C. Also, VPC-A is connected via HA VPN to VPC-C (with BGP sessions in place). Therefore, VPC-A receives the Subnet-C1 advertisement from both the VPC-B (dynamic peered route) and from the cloud router where the VPN gateway is connected (VPC-A). Now consider a VM located in Subnet-A1 that aims at connecting to some other VM located into Subnet-C1: which route will it take?
Neither of the received routes classify within the list of route type precedence (as both are dynamic) and both the routes target the same CIDR (i.e. they have same specificity). The traveller is confused and needs more help determining which of the two routes would be taken!
Third round: Route’s next-hop locality
When a clear decision cannot be made by only looking at the route type and at the route specificity, GCP evaluates the Next-Hop Locality of the contending routing rules. Simply put, GCP prefers the routes that have the most local next-hop for the current VPC. That is, when two warriors of the same specialty fight against each other, the one that knows better the battlefield will win the battle.
In order to determine the locality of the next hop, we need to refer to the official routing documentation here.
In our previous example, the packet will take the VPN route instead of the peering route. This happens because the route’s next hop from VPC-A to Subnet-C1 is the VPN gateway, which lives in VPC-A. The other contending route is a dynamic peered route, therefore the corresponding next hop is located into a peered network, which is not local to VPC-A. This explains why the traffic flows through the VPN tunnel between VPC-A and VPC-C and does not traverse the peering between VPA-A and VPC-B.
On a personal note, the route’s next-hop locality was hard to swallow: I’ve always thought that peering would have had some sort of precedence above VPNs or Interconnects, but that’s untrue. This aspect has great importance especially when dealing with interconnects. Like VPNs, they have a next-hop within the VPC they are attested into, and there is little we can do to manually override this behavior (use PBR or use custom, more specific routes).
You might be thinking, is there still any corner case in which the locality does not help discriminating against all the remaining conflicting cases? The answer is… yes, there are. Let’s consider the following example (as an evolution of the previous network topology.
We created a classic static route into VPC-A for Subnet-C1 with a local next-hop (10.0.0.20). This route conflicts with both the previous ones (dynamic peered route and BGP route from HA VPN). More specifically, this route has a VPC-local next hop, so GCP needs to evaluate something else to determine which route will be preferred.
Fourth round: Route priority
Whenever two or more classic-static/dynamic/peered-dynamic routes with same destination CIDR and same locality compete, Google Cloud breaks the tie by considering the priority associated with each route. In this case, the lowest priority absolute value has the greatest importance.
In our previous example, the classic static route shares the same target CIDR while having a Next-Hop within the VPC-A, as does the VPN tunnel. Packets flowing from a VM in Subnet-A1 and headed to Subnet-C1 will take the static route only if the configured priority value is smaller than the priority value associated with the VPN tunnel route. More specifically, by setting the custom route with a priority of 100, we’ll see the packets headed to 10.0.2.0/24 being routed to 10.0.0.20 instance. On the other hand, if we set the static route priority to 2000, we’ll see packets flowing via the VPN tunnel.
I see priority as the experience each warrior has. Between two warriors competing in the same weapon specialty and belonging to the same kingdom, the one that has more experience will win the battle.
The fourth round was, for me, another epiphany. For a long time I’ve been under the impression that route priority had great importance and could somehow override the behavior of the locality rules. As you see, that’s not the case. This is something network designers must keep in mind when designing complex topologies in GCP.
We are not done quite yet. As you can imagine, we can further evolve our initial example topology by manually adjusting static route’s priority to match the VPN tunnel route’s priorioty. That puts our traveller (IP Packet) into another ambiguous situation… which route will he ever take?
The Tiebreaker’s Tale: When Routes are Equally Matched
In rare cases, routes might be evenly matched in terms of type, specificity, next-hop locality and even custom priority. In such scenarios, GCP employs additional tiebreakers, as described in point 7 in Routing Order Documentation.
First choice: custom static routes (with a running next hop instance) or IKE SA established next hop Classic VPN tunnel;
Second choice: custom dynamic routes learned from any BGP session of any Cloud Router (including BGP sessions);
Third choice: a single custom static route with an internal passthrough Network Load Balancer next hop;
Fourth choice: a custom static route using the default-internet-gateway next hop.
That answers our question: the packet will prefer the static route if that matches exactly the priority of the BGP HA VPN.
In other words, when two warriors compete against each other in the same weapon specialty, with same knowledge of the field and with same level of experience, then their destiny is decided by written faith. I’m really running out of metaphors here, so let’s just say static routes or classic vpn routes win over the others, including dynamic routes.
Navigating the Labyrinth with Confidence
Eventually, our traveller has finally reached their destination, while we’ve consolidated our knowledge about the routing logic under the Google Cloud’s hood. To be honest, there still are some other corner cases that we did not cover. For instance, what happens if a static route has a next-hop that is turned off ? That and other deeper details are explained in the official GCP Routing Documentation, our routing Holy Grail.
Nevertheless, even after reading the entire page over and over, I felt like I needed some more usable and immediate way for handling complex scenarios. That’s why I’ve ended up drawing the following decision tree diagram that I consult whenever I have doubts regarding routing in GCP.
Conclusions
Mastering GCP’s routing precedence empowers us to architect robust and secure networks. By understanding how route types, CIDR prefixes, and priority interact, we can better steer our traffic through the intricate net of peerings, VPNs, and interconnects.
So, the next time you encounter a routing conundrum, remember this knowledge is your compass, guiding you towards optimal network performance and predictable traffic flow.
