Architecting Secure Network Topology for a Multi-Entity Organization

11 min readJun 2, 2024

Setting up a network topology for an organization with multiple entities can be a complex task, especially when the requirement is to provide isolated networks with a shared AWS Network Firewall. In this blog, we will delve into the technical details of how we achieved this setup and the considerations that were taken into account.

To begin with, it is essential to understand the concept of network topology. Network topology refers to the arrangement of the network elements for an environment either on cloud or in data centers network. In the context of setting up isolated networks for multiple entities within an organization, the topology plays a crucial role in ensuring secure and efficient communication.

Concepts

The following are the key concepts for designing and implementing network topology on AWS :

1. VPC

With Amazon Virtual Private Cloud (Amazon VPC), you can launch AWS resources in a logically isolated virtual network that you’ve defined. This virtual network closely resembles a traditional network that you’d operate in your own data center, with the benefits of using the scalable infrastructure of AWS.

2. Transit Gateway

AWS Transit Gateway connects your Amazon Virtual Private Clouds (VPCs) and on-premises networks through a central hub. This connection simplifies your network and puts an end to complex peering relationships. Transit Gateway acts as a highly scalable cloud router — each new connection is made only once.

3. Transit Gateway Route Tables

Transit Gateway route tables are used to control the routing of network traffic within an AWS Transit Gateway. These route tables contain rules that determine how network traffic is directed between the various attachments connected to the Transit Gateway, such as VPCs, VPN connections, and Direct Connect gateways. By configuring the route tables, you can specify the paths that traffic should take to reach its destination, allowing for efficient and flexible network routing within the AWS environment.

4. Transit Gateway Attachments

Transit Gateway can have following two types of attachments

A VPC attachment for AWS Transit Gateway is a connection between a Virtual Private Cloud (VPC) and the AWS Transit Gateway. It allows the VPC to communicate with other VPCs, on-premises networks, and other network resources that are connected to the Transit Gateway. This attachment simplifies network management by providing a centralized point for routing and allows for efficient interconnectivity between different network environments within AWS.
A VPN attachment for Transit Gateway is a connection that allows secure communication between an AWS Transit Gateway and an on-premises network or another remote network over the Internet using Virtual Private Network (VPN) technology

5. Peering Connections

Peering connections on AWS enable the interconnection of Virtual Private Clouds (VPCs) within the same or different AWS accounts or Transit Gateways (TGWs). This allows the VPCs or TGWs to communicate with each other as if they were on the same network. Peering connections can be established between VPCs or TGWs in the same region or in different regions, providing a flexible and scalable way to create network connectivity between different parts of an AWS infrastructure.

5. VPN Tunnels

A VPN tunnel is a secure, encrypted connection established over a public network, typically the internet, to ensure the confidentiality and integrity of data transmitted between two endpoints. It provides a secure “tunnel” for data to travel through, protecting it from unauthorized access or tampering. VPN tunnels are commonly used to establish secure connections between remote users, branch offices, or between a user and a corporate network, allowing for private and secure communication over potentially insecure networks.

6. Direct Connect

Direct Connect is an AWS service that provides a dedicated network connection between an on-premises data center or corporate network and the AWS cloud. It allows for a private, high-bandwidth, low-latency connection that bypasses the public internet, providing a more consistent and reliable network experience compared to standard internet connections.

Prerequisites

This blog post assumes that you have basic understanding deployment models of AWS Network Firewall. If not then I would recommend to get the understanding here

Requirement

This customer operates a group entity with ownership of multiple entities across various countries. As they embarked on their cloud adoption journey, they established the groundwork with AWS Landing Zone. This foundation encompassed the setup of their network topology, a crucial aspect of which was the implementation of a shared firewall for traffic inspection across all entities. However, a key requirement for this customer was to ensure that the traffic between different entities remains isolated, preventing any accidental connections between them.

Proposed Solution

High Level Design

As shown in the above diagram, as part of AWS Landing Zone setup, we designed OU structure in such a way that

All AWS accounts of an entity are grouped together in one Organization Unit (OU)
Each entity has its own transit gateway, which is shared only with a specific OU associated with the respective entity. e.g. TGW of entity 1 is shared only with entity 1 OU, TGW of entity 2 is shared only with entity 2 OU and likewise.

Due to the customer’s requirement for centralized network inspection to enable their global security team to have control over network security, we suggested following two options

Create AWS Network Firewall for each entity and manage network firewall rule groups centrally using AWS Firewall Manager service
Create a Shared Network account for all the entities. Create a transit gateway in this account and establish peering connection with all other transit gateways of respective entities.

Since this customer was very cost sensitive and at the same time they wanted to meet the requirements, customer chose option #2 out of the two options that we proposed.

Some important considerations in designing this solution included:

Having a single view of network traffic and control using a centralized AWS Network Firewall.
Cost optimization with a shared AWS network firewall, which was deployed in 3 Availability Zones (AZs).
We acknowledge that data transfer through the transit gateway incurs charges on AWS. Despite the additional cost associated with each network request flowing through at least two transit gateways, the customer accepted this cost over the benefits of #1 and #2.

Low Level Design

As shown in the above low level design of network architecture, we will understand how entity 1 is leveraging shared network components such as AWS Network firewall in Shared OU while maintaining complete isolation of entity 1 traffic from rest of the entities and their on-premise network.

This architecture covers following requirements

Inspect inter vpc traffic centrally with shared AWS Network firewall
Inspect on-premise traffic centrally with shared AWS Network firewall
Inspect egress internet traffic centrally with shared AWS Network firewall and central egress VPC
Isolate entity 1 traffic from rest of the entities and their on-premise network

Let’s understand traffic flow for all the above use cases one by one

Inspect inter vpc traffic centrally with shared AWS Network firewall

Let’s assume that an EC2 instance in non-production VPC A requires to connect EC2 instance in non-production VPC B. So the traffic flow for this connectivity is as below

EC2 instance in VPC A initiates request to connect to EC2 instance in VPC B
Traffic first arrives at VPC A route table and with default route to transit gateway, this request reach transit gateway.
VPC A is associated with non-production TGW route table and hence next routing decision will be taken by routes configured on this route table
With default route to TGW peering connection, this request reach to transit gateway in Shared OU
TGW peering connection in Shared OU is associated a TGW Peering route table which decides to send this request further to inspection VPC using it’s default route
AWS Network firewall is deployed in Inspection VPC. VPC route table associated with Inspection VPC’s TGW attachment sends this request further to gateway load balancer endpoints of AWS Network firewall
AWS Network firewall further evaluate the traffic based on it’s firewall rules and assuming firewall rule allow traffic from VPC A to VPC B, firewall will allow the traffic.
This request is further sent to Inspection VPC TGW route table with which Inspection VPC is associated it. Based on the routes, request is further sent to TGW peering connection of entity 1 TGW
TGW peering transit gateway route table of entity 1 has propagated routes for all the respective VPCs so with this propagated route, request reach to VPC B
EC2 instance in VPC B process the request and initiate response back to the originating IP in VPC A.
Response traffic follows the same route as explained for request traffic and reach back to originating IP in VPC A

Inspect on-premise traffic centrally with shared AWS Network firewall

Let’s assume that an EC2 instance in non-production VPC A requires to connect to on-premise server. So the traffic flow for this connectivity is as below

EC2 instance in VPC A initiates request to connect to on-premise server
Traffic first arrives at VPC A route table and with default route to transit gateway, this request reach transit gateway.
VPC A is associated with non-production TGW route table and hence next routing decision will be taken by routes configured on this route table
With default route to TGW peering connection, this request reach to transit gateway in Shared OU
TGW peering connection in Shared OU is associated a TGW Peering route table which decides to send this request further to inspection VPC using it’s default route
AWS Network firewall is deployed in Inspection VPC. VPC route table associated with Inspection VPC’s TGW attachment sends this request further to gateway load balancer endpoints of AWS Network firewall
AWS Network firewall further evaluate the traffic based on it’s firewall rules and assuming firewall rule allow traffic from VPC A to on-premise, firewall will allow the traffic.
This request is further sent to Inspection VPC TGW route table with which Inspection VPC is associated it. Based on the routes, request is further sent to TGW peering connection of entity 1 TGW
TGW peering transit gateway route table of entity 1 has propagated routes for VPN attachment so with this propagated route, request reach to on-premise server through VPN tunnel
On-premise server process the request and initiate response back to the originating IP in VPC A.
Response traffic arrives at TGW in entity 1 through VPN tunnel and routed back to inspection VPC through TGW route table for VPN.
AWS Network firewall inspect the return traffic and send response back to originating IP in VPC A of entity 1 as explained in the previous packet flow

Inspect egress internet traffic centrally with shared AWS Network firewall and central egress VPC

Let’s assume that an EC2 instance in non-production VPC A requires to connect to an internet resource. So the traffic flow for this connectivity is as below

EC2 instance in VPC A initiates request to a resource accessible over internet
Traffic first arrives at VPC A route table and with default route to transit gateway, this request reach transit gateway.
VPC A is associated with non-production TGW route table and hence next routing decision will be taken by routes configured on this route table
With default route to TGW peering connection, this request reach to transit gateway in Shared OU
TGW peering connection in Shared OU is associated a TGW Peering route table which decides to send this request further to inspection VPC using it’s default route
AWS Network firewall is deployed in Inspection VPC. VPC route table associated with Inspection VPC’s TGW attachment sends this request further to gateway load balancer endpoints of AWS Network firewall
AWS Network firewall further evaluate the traffic based on it’s firewall rules and assuming firewall rule allow traffic from VPC A to internet, firewall will allow the traffic.
This request is further sent to Inspection VPC TGW route table with which Inspection VPC is associated it. Based on the routes, request is further sent to TGW attachment of Central Egress VPC
Traffic arrives at TGW route table of Central Egress VPC and based on the VPC route table associated with TGW attachment, traffic is further sent to NAT Gateway
NAT Gateway further send this request over internet through internet gateway
Internet resource process the request and initiate response back to the originating public IP associated with NAT gateway.
Response traffic arrives at internet gateway (IGW), and further reach to NAT gateway. VPC route table associated with NAT Gateway further send this response back to TGW
TGW route table associated with Central Egress VPC further send this response traffic back to inspection VPC for inspection
AWS Network firewall inspect the response and send back to originating IP in VPC A of entity 1 as explained in the previous packet flow

Isolate entity 1 traffic from rest of the entities and their on-premise network

Now it is important to understand how this architecture is helping to isolate traffic between entities even if shared firewall is used for traffic inspection. Let’s understand this with different use cases

Non-Prod VPC initiates traffic to Production VPC

EC2 instance in VPC A initiates request to a resource in Production VPC of entity 1
Traffic first arrives at VPC A route table and with default route to transit gateway, this request reach transit gateway.
VPC A is associated with non-production TGW route table and hence next routing decision will be taken by routes configured on this route table
With blackhole entry to production CIDR, this request is dropped at TGW route table itself and the connectivity is restricted

Entity 1 VPC initiates traffic to entity 2 VPC

EC2 instance in VPC A initiates request to a resource in entity 2 VPC
Traffic first arrives at VPC A route table and with default route to transit gateway, this request reach transit gateway.
VPC A is associated with non-production TGW route table and hence next routing decision will be taken by routes configured on this route table
With blackhole entry to entity 2 CIDR, this request is dropped at TGW route table itself and the connectivity is restricted

Entity 1 VPC initiates traffic to entity 2 data center

EC2 instance in VPC A initiates request to a resource in data center of entity 2
Traffic first arrives at VPC A route table and with default route to transit gateway, this request reach transit gateway.
VPC A is associated with non-production TGW route table and hence next routing decision will be taken by routes configured on this route table
With blackhole entry to on-premise CIDRs of other entities, this request is dropped at TGW route table itself and the connectivity is restricted

Conclusion

In this article, we discusses the approach used to set up a secure network topology for a multi-entity organization on AWS. It covers the key concepts involved, such as VPCs, Transit Gateways, and Network Firewall, and how they were leveraged to create isolated networks with a shared firewall for centralized traffic inspection.

Key Points

Explained the key concepts of VPC, Transit Gateway, Transit Gateway Route Tables, Transit Gateway Attachments, Peering Connections, and VPN Tunnels.
Described the high-level design and requirements, including the need for centralized network inspection and cost optimization.
Detailed the low-level design and packet flow for inspecting inter-VPC traffic, on-premises traffic, and egress internet traffic through the shared AWS Network Firewall.
Explained how the architecture ensures isolation of traffic between different entities, even with the use of a shared firewall.