Set up Kubernetes clusters in multiple clouds
Why would someone want to deploy Kubernetes clusters to AWS, GCP and Azure simultaneously? This sounds needlessly complicated. The SDA SE offers a full development solution for customers, including, e.g., CI/CD, monitoring or security tools, where customers are free to choose at which cloud provider their environment should be provisioned. Thus, there is a demand to
- have a solution that is as cloud-agnostic as possible
- provide a network concept for admins/DevOps
- provide network access to customers
As we don’t want to provide our own solution of a Kubernetes cluster, we use the managed Kubernetes cluster solution of the respective cloud provider, i.e., EKS, GKE, and AKS. This is the cloud-specific part. On top of this, we deploy our tool stack in a cloud-agnostic manner. Moreover, the Kubernetes clusters shall provide private-only access, and everything should be provided as IaC. In our case, IaC is accomplished through the HCL language and infrastructure is provisioned by Terraform.
This series consists in total of four articles. This article describes the network concepts and how the three cloud providers, AWS, Azure and GCP, can be interconnected using VPN site-2-site tunnels. The following articles will describe the particular setup for the specific cloud providers, including Kubernetes cluster and private/public DNS solution.
Setting the stage
As prerequisites, we state that a Kubernetes cluster master API shall only be accessible from the internal network. Further, services are categorized as “internal” and “external”, respectively. The former are exposed through internal load balancers. Those are reachable from within the internal network, only. The latter services are reachable from the world utilizing internet-facing load balancers. The DNS A-records for internal services are created in private DNS zones, whereas those associated with external services are publicly resolvable. More details will follow later. Last but not least, different Kubernetes clusters should not be allowed to reach each other.
As the three cloud providers have slightly different networking concepts, we mean by a VPC a vNet in Azure, and a VPC in AWS/GCP, respectively. Although GCP VPCs are not assigned any CIDR ranges, in the context required for this article, we can think of a VPC in GCP as having a CIDR range assigned. Again, details will follow in upcoming articles.
Having the above setting in mind, we decided to set up a mostly star-shaped hub-and-spoke VPC topology, where the center VPC will forward traffic to the leaf VPCs dependent on permissions provided by our IdP (Identity Provider). To be more precise, we have a star-shaped VPC at each cloud provider and establish site-to-site VPN connections between the center VPCs. In our case, AWS is the “primary” provider. This means, we define forwarding rules, DNS forwarders, and top-level DNS resolvers here. For the sake of simplicity, we want that all the subnets are routable, thus we define non-overlapping CIDR ranges for each VPC and assign the following CIDR ranges to each cloud provider:
- AWS: 10.0.0.0/10
- GCP: 10.64.0.0/10
- Azure: 10.128.0.0/10
For each cloud provider, the first /16 CIDR subrange will be reserved for the respective center VPC, i.e.:
- AWS: 10.0.0.0/16
- GCP: 10.64.0.0/16
- Azure: 10.128.0.0/16
AWS
In AWS each EKS cluster have a one-to-one association with a VPC and leaf VPCs are connected to the center VPC by means of a transit gateway. An AWS account can host more than one VPC, however, customer VPCs are in a different account than the center VPC, the latter one being a pure jump VPC. Thus, we need to provision the following topology:
The steps required to be performed are:
- create the center VPC
- create a transit gateway, a RAM share and associate both
- create a leaf VPC
- invite and accept the RAM share
- attach leaf VPC to gateway
- establish routes from private subnets of leaf VPC to center VPC
- establish routes from private subnets of center VPC to leaf VPC
First, we create the center VPC:
locals {
availability_zones = ["eu-central-1a", "eu-central-1b", "eu-central-1c"]
public_subnet_names = ["public-1a", "public-1b", "public-1c"]
private_subnet_names = ["private-1a", "private-1b", "private-1c"]
center_vpc_private_subnet_ids = [for i in local.private_subnet_names : module.center_vpc_cidr.network_cidr_blocks[i]]
aws_bgp_asn = 64512
aws_cidr_range = cidrsubnet("10.0.0.0/8", 2, 0)
}module "center_vpc_cidr" {
source = "registry.terraform.io/hashicorp/subnets/cidr" base_cidr_block = cidrsubnet(local.aws_cidr_range, 8, 0)
networks = concat(
[for i in local.public_subnet_names : { name = i, new_bits = 4 }],
[{ name = "not-associated", new_bits = 4 }],
[for i in local.public_subnet_names : { name = i, new_bits = 2 }],
)
}module "center_vpc" {
providers = {
aws = aws.center_vpc
} source = "terraform-aws-modules/vpc/aws" name = "center-vpc"
cidr = module.center_vpc_cidr.base_cidr_block azs = local.availability_zones
private_subnets = local.center_vpc_private_subnet_ids
public_subnets = [for i in local.public_subnet_names : module.center_vpc_cidr.network_cidr_blocks[i]] enable_nat_gateway = true
enable_vpn_gateway = true tags = local.vpc_tags
}For the second step, we have the following Terraform description:
resource "aws_ec2_transit_gateway" "gateway" {
provider = aws.center_vpc description = "This TG connects this VPN VPC with all AWS and GCP VPCs, and Azure vNETs"
amazon_side_asn = local.aws_bgp_asn
auto_accept_shared_attachments = "enable"
default_route_table_association = "enable"
default_route_table_propagation = "enable"
dns_support = "enable"
vpn_ecmp_support = "enable" tags = local.vpc_tags
}resource "aws_ram_resource_share" "gateway" {
provider = aws.center_vpc name = var.name
allow_external_principals = true tags = local.vpc_tags
}resource "aws_ram_resource_association" "gateway" {
provider = aws.center_vpc resource_arn = aws_ec2_transit_gateway.gateway.arn
resource_share_arn = aws_ram_resource_share.gateway.id
}As a last step, we create the leaf VPC, invite/accept RAM share and associate leaf VPC, and create routes:
module "leaf_vpc_cidr" {
source = "registry.terraform.io/hashicorp/subnets/cidr" base_cidr_block = cidrsubnet(local.aws_cidr_range, 8, 1)
networks = concat(
[for i in local.public_subnet_names : { name = i, new_bits = 4 }],
[{ name = "not-associated", new_bits = 4 }],
[for i in local.public_subnet_names : { name = i, new_bits = 2 }],
)
}module "leaf_vpc" {
providers = {
aws = aws.leaf_vpc
} source = "terraform-aws-modules/vpc/aws" name = "center-vpc"
cidr = module.center_vpc_cidr.base_cidr_block azs = local.availability_zones
private_subnets = [for i in local.private_subnet_names : module.leaf_vpc_cidr.network_cidr_blocks[i]]
public_subnets = [for i in local.public_subnet_names : module.leaf_vpc_cidr.network_cidr_blocks[i]] enable_nat_gateway = true
enable_vpn_gateway = true tags = local.vpc_tags
}resource "aws_ram_principal_association" "leaf_invite" {
provider = aws.center_vpc principal = var.leaf_account_id
resource_share_arn = aws_ram_resource_share.gateway.arn
}resource "aws_ram_resource_share_accepter" "leaf_accept" {
provider = aws.leaf_vpc share_arn = aws_ram_principal_association.leaf_invite.resource_share_arn
}resource "aws_ec2_transit_gateway_vpc_attachment" "leaf_vpc" {
provider = aws.leaf_vpc subnet_ids = module.leaf_vpc.private_subnets
transit_gateway_id = aws_ec2_transit_gateway.vpn_gateway.id
vpc_id = module.leaf_vpc.vpc_id dns_support = "enable"
transit_gateway_default_route_table_association = true
transit_gateway_default_route_table_propagation = true tags = local.vpc_tags depends_on = [aws_ram_principal_association.leaf_invite]
}resource "aws_route" "leaf_to_center" {
provider = aws.leaf_vpc for_each = toset(module.leaf_vpc.private_route_table_ids) route_table_id = each.value
destination_cidr_block = module.center_vpc_cidr.base_cidr_block
transit_gateway_id = aws_ec2_transit_gateway.gateway.id
}resource "aws_route" "center_to_leaf" {
provider = aws.center_vpc for_each = toset(module.center_vpc.private_route_table_ids) route_table_id = each.value
destination_cidr_block = module.leaf_vpc_cidr.base_cidr_block
transit_gateway_id = aws_ec2_transit_gateway.gateway.id
}VPN client reaches the stage
At this point we have two VPCs that forward traffic from within their private subnets to the other VPC. We have not said a single word on security groups, and don’t plan to do so at this point, since this topic is tightly connected to applications. However, we need to get into the center VPC from a client machine. This can be performed using, e.g., AWS VPN client.
As we want to access the client through an address placed in our subdomain, we need to issue a certificate. There are several methods to do so. Here, we use the AWS-provided ACME challenge to issue a wildcard certificate:
locals {
domain_name = "your.domain.goes.here"
}resource "aws_acm_certificate" "vpn_client" {
provider = aws.center_vpc domain_name = "*.vpn.${local.domain_name}"
validation_method = "DNS"
lifecycle {
create_before_destroy = true
} tags = local.vpc_tags
}resource "aws_acm_certificate_validation" "vpn_client" {
provider = aws.center_vpc certificate_arn = aws_acm_certificate.vpn_client.arn
validation_record_fqdns = [aws_route53_record.vpn_client_validation.fqdn] timeouts {
create = "1m"
}
depends_on = [aws_route53_record.vpn_client_validation]
}resource "aws_route53_record" "vpn_client_validation" {
provider = aws.center_vpc zone_id = aws_route53_zone.this__sda_se_io.zone_id
name = tolist(aws_acm_certificate.vpn_client.domain_validation_options)[0].resource_record_name
type = tolist(aws_acm_certificate.vpn_client.domain_validation_options)[0].resource_record_type
ttl = "60" records = [tolist(aws_acm_certificate.vpn_client.domain_validation_options)[0].resource_record_value]}resource "aws_route53_record" "vpn_client_wildcard" {
provider = aws.center_vpc zone_id = aws_route53_zone.this__sda_se_io.zone_id
name = "*.vpn.${local.domain_name}"
type = "CNAME"
ttl = "60" records = ["vpn.${local.domain_name}"]
}Now that we have a valid certificate at hand, we can eventually create the VPN client endpoint and a CNAME for it:
locals {
idp_group_ids = ["your-group-ids"]
# A CIDR subrange not occupied by any subnet of center VPC, at least /22. In the above setup we have a free /20 left.
client_cidr_block = cidrsubnet(module.center_vpc_cidr.network_cidr_blocks["not-associated"], 2, 0)
access_to_vpn_client = ["0.0.0.0/0"]
}resource "aws_ec2_client_vpn_endpoint" "vpn_client" {
provider = aws.center_vpc description = "Client-VPN"
server_certificate_arn = aws_acm_certificate_validation.vpn_client.certificate_arn
client_cidr_block = local.client_cidr_block
split_tunnel = true
self_service_portal = "enabled"
# use VPCs DNS servers for DNS resolution on client machines
dns_servers = [cidrhost(module.center_vpc_cidr.base_cidr_block, 2)]
security_group_ids = [aws_security_group.vpn_client_access.id] authentication_options {
type = "federated-authentication"
saml_provider_arn = aws_iam_saml_provider.idp_saml.arn # connect your IdP here
} connection_log_options {
enabled = true
cloudwatch_log_group = aws_cloudwatch_log_group.vpn_client.name
cloudwatch_log_stream = aws_cloudwatch_log_stream.vpn_client.name
} tags = local.vpc_tags
}resource "aws_security_group" "vpn_client_access" {
provider = aws.center_vpc vpc_id = module.center_vpc.vpc_id
name = "client-vpn"
description = "Client VPN" ingress {
description = "Incoming VPN connections"
from_port = 443
protocol = "UDP"
to_port = 443
cidr_blocks = local.access_to_vpn_client
} egress {
description = "Allow all egress"
from_port = 0
protocol = "-1"
to_port = 0
cidr_blocks = ["0.0.0.0/0"]
} tags = local.vpc_tags
}resource "aws_cloudwatch_log_group" "vpn_client" {
provider = aws.center_vpc #checkov:skip=CKV_AWS_158: for production, enable encryption
name = "/aws/vpn/${var.name}/logs"
retention_in_days = var.logs_retention tags = local.vpc_tags
}resource "aws_cloudwatch_log_stream" "vpn_client" {
provider = aws.center_vpc name = "vpn-usage"
log_group_name = aws_cloudwatch_log_group.vpn_client.name
}resource "aws_iam_saml_provider" "idp_saml" {
provider = aws.center_vpc name = "vpn-client"
saml_metadata_document = file("saml_metadata.xml") tags = local.vpc_tags
}resource "aws_route53_record" "vpn_client" {
provider = aws.center_vpc zone_id = aws_route53_zone.this__sda_se_io.zone_id
name = "vpn.${local.domain_name}"
type = "CNAME"
ttl = "60" records = [replace("your-client.${aws_ec2_client_vpn_endpoint.vpn_client.dns_name}", "*.", "")]
}For production environments, log group encryption should be turned on. Moreover, it might be desired to restrict access to the VPN client endpoint instead of having it open for the whole world.
Now that the client has been created, it has to be granted access to center VPC and has to be associated with required subnets from within it:
resource "aws_ec2_client_vpn_authorization_rule" "center_vpc" {
provider = aws.center_vpc client_vpn_endpoint_id = aws_ec2_client_vpn_endpoint.vpn_client.id
target_network_cidr = module.center_vpc_cidr.base_cidr_block
authorize_all_groups = true
}resource "aws_ec2_client_vpn_network_association" "center_vpc" {
provider = aws.center_vpc for_each = toset(local.center_vpc_private_subnet_ids)
client_vpn_endpoint_id = aws_ec2_client_vpn_endpoint.vpn_client.id
subnet_id = each.key
}As a last step, authorize VPN client to leaf VPC and define routes to leaf private subnets:
resource "aws_ec2_client_vpn_authorization_rule" "leaf_vpc" {
provider = aws.leaf_vpc for_each = toset(local.idp_group_ids) client_vpn_endpoint_id = aws_ec2_client_vpn_endpoint.vpn_client.id
target_network_cidr = module.leaf_vpc_cidr.base_cidr_block
access_group_id = each.value
}resource "aws_ec2_client_vpn_route" "vpn_routes" {
provider = aws.center_vpc for_each = toset(local.center_vpc_private_subnet_ids) client_vpn_endpoint_id = aws_ec2_client_vpn_endpoint.vpn_client.id
destination_cidr_block = module.leaf_vpc_cidr.base_cidr_block
target_vpc_subnet_id = each.value
}That’s all. Summing up we have:
- created a center VPC with 3 private (/18) and 3 public subnets (/20)
- created a VPN client and associated it with the 3 private subnets of center VPC
- allowed VPN clients communication with whole center VPC
- allowed VPN clients with certain groups communication with whole leaf VPC
- created routing rules (subnet and client VPN) from private subnets of center VPC to whole CIDR of leaf VPC
- created routing rules (subnet and client VPN) from private subnets of leaf VPC to whole CIDR of center VPC
GCP
Dependent on requirements, there are several possibilities to connect different clouds. Here, we will use a site-2-site VPN tunnel using BGP protocol. This is accomplished using customer gateway on AWS site and HA vpn gateway on GCP site, as described in this tutorial. Moreover, we need to include routing rules for center VPC and VPN client in AWS, and configure a cloud router and firewall rules in GCP. The situation is depicted below.
As a first step, we need to create the center VPC in GCP, configure firewall to allow communication from AWS, and provision the VPN endpoint:
locals {
gcp_region = "europe-west3"
gcp_default_zone = "europe-west3-c"
gcp_center_project_id = "shared-vpc"
gcp_center_network_name = "transfer-vpc"
gcp_center_subnet_name = "${local.gcp_center_network_name}-subnet"
gcp_bgp_asn = 64514
gcp_cidr_range = cidrsubnet("10.0.0.0/8", 2, 1)
}module "gcp_center_vpn_network" {
source = "registry.terraform.io/terraform-google-modules/network/google"
version = "5.0.0" project_id = local.gcp_center_project_id
network_name = local.gcp_center_network_name
description = "Shared VPC"
routing_mode = "GLOBAL" shared_vpc_host = true subnets = [
{
subnet_name = local.gcp_center_subnet_name
subnet_ip = cidrsubnet(local.gcp_cidr_range, 8, 0)
subnet_region = local.gcp_region
subnet_private_access = "true"
},
]
}resource "google_compute_firewall" "allow_traffic_from_aws" {
project = local.gcp_center_project_id
name = "allow-from-aws"
network = module.gcp_center_vpn_network.network_name
description = "Allows traffic from AWS"
priority = 200 allow {
protocol = "icmp"
} allow {
protocol = "tcp"
ports = ["443"]
} source_ranges = [module.center_vpc_cidr.base_cidr_block]
direction = "INGRESS"
}resource "google_compute_firewall" "allow_all_outbound" {
project = local.gcp_center_project_id
name = "allow-all-outbound"
network = module.gcp_center_vpn_network.network_name
description = "Allows all outbound traffic"
priority = 200 allow {
protocol = "all"
} destination_ranges = ["0.0.0.0/0"]
direction = "EGRESS"
}resource "google_compute_ha_vpn_gateway" "transfer_vpn_gcp_aws" {
project = local.gcp_center_project_id
region = local.gcp_region
name = "ha-${local.gcp_center_network_name}"
network = module.gcp_center_vpn_network.network_id
}resource "google_compute_router" "transfer_vpn_gcp_aws" {
project = local.gcp_center_project_id
name = local.gcp_center_network_name
network = module.gcp_center_vpn_network.network_name
bgp {
asn = local.gcp_bgp_asn
advertise_mode = "CUSTOM"
advertised_groups = ["ALL_SUBNETS"] advertised_ip_ranges {
range = local.gcp_cidr_range
description = "All GCP"
}
}
}Next, the customer gateway needs to be connected to the above provided VPN gateway. Since this is a HA gateway, we need to create two customer gateways. Thus, we will have a total of four VPN connections, eventually. For each of the VPN connections, a pre-shared key is required. This is just a random password with a restricted set of special characters. Moreover, as described here, we need to select a combination of just single Phase 1 and Phase 2 values, thus all tunnel1_ and tunnel2_ prefixed list-type values have a length one in the following aws_vpn_connection resource.
locals {
gw_names = ["gw1", "gw2"]
gcp_gateway_options = {
for i in range(length(local.gw_names)) : local.gw_names[i] => {
tunnel1_inside_cidr = cidrsubnet("169.254.0.0/16", 14, 6+2*i)
tunnel2_inside_cidr = cidrsubnet("169.254.0.0/16", 14, 7+2*i)
gcp_gateway_ip = google_compute_ha_vpn_gateway.transfer_vpn_gcp_aws.vpn_interfaces[i].ip_address
}
}
}resource "random_password" "transfer_aws_gcp_psk_1" {
for_each = local.gcp_gateway_options length = 63
lower = true
upper = true
number = true
special = true
override_special = "._"
}resource "random_password" "transfer_aws_gcp_psk_2" {
for_each = local.gcp_gateway_options length = 63
lower = true
upper = true
number = true
special = true
override_special = "._"
}resource "aws_customer_gateway" "transfer_aws_gcp" {
provider = aws.center_vpc for_each = local.gcp_gateway_options bgp_asn = local.gcp_bgp_asn
ip_address = each.value.gcp_gateway_ip
type = "ipsec.1" tags = local.gcp_transfer_tags
}resource "aws_vpn_connection" "transfer_aws_gcp" {
provider = aws.center_vpc for_each = local.gcp_gateway_options customer_gateway_id = aws_customer_gateway.transfer_aws_gcp[each.key].id
type = aws_customer_gateway.transfer_aws_gcp[each.key].type
transit_gateway_id = aws_ec2_transit_gateway.vpn_gateway.id tunnel1_inside_cidr = each.value.tunnel1_inside_cidr
# avoid illegal 0 at beginning
tunnel1_preshared_key = "a${random_password.transfer_aws_gcp_psk_1[each.key].result}"
tunnel1_ike_versions = ["ikev2"]
tunnel1_phase1_dh_group_numbers = [20]
tunnel1_phase1_encryption_algorithms = ["AES256"]
tunnel1_phase1_integrity_algorithms = ["SHA2-512"]
tunnel1_phase2_dh_group_numbers = [22]
tunnel1_phase2_encryption_algorithms = ["AES256"]
tunnel1_phase2_integrity_algorithms = ["SHA2-512"] tunnel2_inside_cidr = each.value.tunnel2_inside_cidr
tunnel2_preshared_key = "a${random_password.transfer_aws_gcp_psk_2[each.key].result}"
tunnel2_ike_versions = ["ikev2"]
tunnel2_phase1_dh_group_numbers = [18]
tunnel2_phase1_encryption_algorithms = ["AES256"]
tunnel2_phase1_integrity_algorithms = ["SHA2-512"]
tunnel2_phase2_dh_group_numbers = [24]
tunnel2_phase2_encryption_algorithms = ["AES256"]
tunnel2_phase2_integrity_algorithms = ["SHA2-512"] tags = local.gcp_transfer_tags
}The customer gateways need a representation in GCP. “FOUR_IPS_REDUNDANCY” need to be specified, since we have a HA gateway, and thus four connections.
locals {
tunnel_names = ["tunnel1", "tunnel2"] aws_gateway_options = merge([
for i in range(length(local.gw_names)) : {
"${local.gw_names[i]}-tunnel1" = {
psk = aws_vpn_connection.transfer_aws_gcp[local.gw_names[i]].tunnel1_preshared_key
inside_cidr = aws_vpn_connection.transfer_aws_gcp[local.gw_names[i]].tunnel1_inside_cidr
customer_inside_cidr = "${aws_vpn_connection.transfer_aws_gcp[local.gw_names[i]].tunnel1_cgw_inside_address}/30"
vpn_gateway = aws_vpn_connection.transfer_aws_gcp[local.gw_names[i]].tunnel1_vgw_inside_address
vpn_outside_ip = aws_vpn_connection.transfer_aws_gcp[local.gw_names[i]].tunnel1_address
external_interface = 0+2*i
interface = i
}
"${local.gw_names[i]}-tunnel2" = {
psk = aws_vpn_connection.transfer_aws_gcp[local.gw_names[i]].tunnel2_preshared_key
inside_cidr = aws_vpn_connection.transfer_aws_gcp[local.gw_names[i]].tunnel2_inside_cidr
customer_inside_cidr = "${aws_vpn_connection.transfer_aws_gcp[local.gw_names[i]].tunnel2_cgw_inside_address}/30"
vpn_gateway = aws_vpn_connection.transfer_aws_gcp[local.gw_names[i]].tunnel2_vgw_inside_address
vpn_outside_ip = aws_vpn_connection.transfer_aws_gcp[local.gw_names[i]].tunnel2_address
external_interface = 1+2*i
interface = i
}
}
]...)
}resource "google_compute_external_vpn_gateway" "transfer_vpn_gcp_aws" {
project = local.gcp_center_project_id
name = "external-gateway-aws"
redundancy_type = "FOUR_IPS_REDUNDANCY"
description = "VPN gateway to AWS" dynamic "interface" {
for_each = local.aws_gateway_options
content {
id = interface.value.external_interface
ip_address = interface.value.vpn_outside_ip
}
}
}With this information, we can finally configure the VPN connections from GCP site, and configure the cloud router instance:
resource "google_compute_vpn_tunnel" "transfer_vpn_gcp_aws" {
for_each = local.aws_gateway_options project = local.gcp_center_project_id
name = "external-gateway-aws-${each.key}"
peer_external_gateway = google_compute_external_vpn_gateway.transfer_vpn_gcp_aws.id
peer_external_gateway_interface = each.value.external_interface
region = local.gcp_region
ike_version = "2"
shared_secret = each.value.psk
router = google_compute_router.transfer_vpn_gcp_aws.id
vpn_gateway = google_compute_ha_vpn_gateway.transfer_vpn_gcp_aws.id
vpn_gateway_interface = each.value.interface
}resource "google_compute_router_interface" "transfer_vpn_gcp_aws" {
for_each = local.aws_gateway_options project = local.gcp_center_project_id
name = "${google_compute_router.transfer_vpn_gcp_aws.name}-${each.key}"
region = local.gcp_region
ip_range = each.value.customer_inside_cidr
router = google_compute_router.transfer_vpn_gcp_aws.name
vpn_tunnel = google_compute_vpn_tunnel.transfer_vpn_gcp_aws[each.key].name
}resource "google_compute_router_peer" "transfer_vpn_gcp_aws" {
for_each = local.aws_gateway_options project = local.gcp_center_project_id
name = "${google_compute_router.transfer_vpn_gcp_aws.name}-${each.key}"
router = google_compute_router.transfer_vpn_gcp_aws.name
region = local.gcp_region
peer_ip_address = each.value.vpn_gateway
peer_asn = local.aws_bgp_asn
interface = google_compute_router_interface.transfer_vpn_gcp_aws[each.key].name
}resource "google_compute_router" "router" {
project = local.gcp_center_project_id
name = "nat-router"
network = module.gcp_center_vpn_network.network_name
region = local.gcp_region
}As a last step, we need to configure AWS route tables and the AWS VPN client to forward traffic to GCP using the newly created tunnels. Dependent on requirements, multiple aws_ec2_client_vpn_authorization_rule s can be configured, each allowing access to just a subrange of the whole google CIDR range "10.64.0.0/10". For sake of simplicity, we define a single rule here.
locals {
idp_gcp_group_ids = ["my-gcp-group-id"]
}resource "aws_route" "transfer_aws_gcp" {
provider = aws.center_vpc for_each = toset(module.center_vpc.private_route_table_ids) route_table_id = each.value
destination_cidr_block = local.gcp_cidr_range
transit_gateway_id = aws_ec2_transit_gateway.vpn_gateway.id
}resource "aws_ec2_client_vpn_route" "transfer_aws_gcp" {
provider = aws.center_vpc for_each = toset(local.center_vpc_private_subnet_ids) client_vpn_endpoint_id = aws_ec2_client_vpn_endpoint.default.id
destination_cidr_block = local.gcp_cidr_range
target_vpc_subnet_id = each.value
description = "Route to GCP CIDR range"
}resource "aws_ec2_client_vpn_authorization_rule" "transfer_aws_gcp" {
provider = aws.center_vpc for_each = toset(local.idp_gcp_group_ids) client_vpn_endpoint_id = aws_ec2_client_vpn_endpoint.vpn_client.id
target_network_cidr = local.gcp_cidr_range
access_group_id = each.value
description = "Authorize AD group ${each.value} to connect to GCP CIDR range"
}Azure
In contrast to GCP, we do not really require BGP protocol for the VPN connection for Azure. Thus, we won’t configure BGP here and, moreover, just a single customer gateway will be provisioned on AWS site. To set up a HA bgp-enabled connection, e.g. this tutorial might be a good starting point. Nevertheless, we have the following situation:
Before we can start provisioning the required gateway, we need to create a center vNet in Azure:
locals {
azure_center_network_name = "center-vpn"
azure_center_subnet_names = ["${local.azure_center_network_name}-subnet", "GatewaySubnet"]
azure_cidr_range = cidrsubnet("10.0.0.0/8", 2, 2)
azure_bgp_asn = 65000
}module "azure_center_vpn_cidr" {
source = "registry.terraform.io/hashicorp/subnets/cidr" base_cidr_block = cidrsubnet(local.azure_cidr_range, 8, 0)
networks = [
{
name = "subnet"
new_bits = 1
},
{
name = "gateway"
new_bits = 11
},
]
}resource "azurerm_resource_group" "transfer_vpn" {
provider = azurerm.center_vpn
location = "westeurope"
name = local.azure_center_network_name
tags = local.azure_transfer_tags
}module "azure_center_vpn_network" {
providers = {
azurerm = azurerm.center_vpn
}
source = "registry.terraform.io/Azure/vnet/azurerm"
version = "2.6.0" resource_group_name = azurerm_resource_group.transfer_vpn.name
vnet_name = local.azure_center_network_name
address_space = [module.azure_center_vpn_cidr.base_cidr_block]
subnet_names = local.azure_center_subnet_names
subnet_prefixes = [
module.azure_center_vpn_cidr.network_cidr_blocks["subnet"],
module.azure_center_vpn_cidr.network_cidr_blocks["gateway"],
] tags = local.azure_transfer_tags # cf. https://github.com/Azure/terraform-azurerm-network/issues/44
depends_on = [azurerm_resource_group.transfer_vpn]
}Analogous to GCP setup, we need the gateway on Azure site, first. This is accomplished with a virtual network gateway
resource "azurerm_public_ip" "transfer_vpn_azure_aws" {
provider = azurerm.center_vpn
name = "transfer-vpn-azure-aws"
resource_group_name = azurerm_resource_group.transfer_vpn.name
location = azurerm_resource_group.transfer_vpn.location allocation_method = "Dynamic"
sku = "Basic" tags = local.azure_transfer_tags
}resource "azurerm_virtual_network_gateway" "transfer_vpn_azure_aws" {
provider = azurerm.center_vpn
name = "transfer-vpn-azure-aws"
resource_group_name = azurerm_resource_group.transfer_vpn.name
location = azurerm_resource_group.transfer_vpn.location type = "Vpn"
vpn_type = "RouteBased" active_active = false
enable_bgp = false
sku = "VpnGw1" ip_configuration {
name = "default"
public_ip_address_id = azurerm_public_ip.transfer_vpn_azure_aws.id
private_ip_address_allocation = "Dynamic"
subnet_id = module.azure_center_vpn_network.vnet_subnets[1]
} tags = local.azure_transfer_tags
}connected to the customer gateway on AWS site
resource "aws_customer_gateway" "transfer_aws_azure" {
provider = aws.center_vpc bgp_asn = local.azure_bgp_asn
ip_address = azurerm_public_ip.transfer_vpn_azure_aws.ip_address
type = "ipsec.1" tags = local.azure_transfer_tags depends_on = [azurerm_virtual_network_gateway.transfer_vpn_azure_aws]
}resource "aws_vpn_gateway" "transfer_aws_azure" {
provider = aws.center_vpc vpc_id = module.center_vpc.vpc_id
tags = local.azure_transfer_tags
}resource "random_password" "transfer_aws_azure_psk_1" {
length = 63
lower = true
upper = true
number = true
special = true
override_special = "._"
}resource "random_password" "transfer_aws_azure_psk_2" {
length = 63
lower = true
upper = true
number = true
special = true
override_special = "._"
}resource "aws_vpn_connection" "transfer_aws_azure" {
provider = aws.center_vpc customer_gateway_id = aws_customer_gateway.transfer_aws_azure.id
type = aws_customer_gateway.transfer_aws_azure.type
vpn_gateway_id = aws_vpn_gateway.transfer_aws_azure.id
static_routes_only = true tunnel1_inside_cidr = local.azure_gateway_options.gw1.tunnel1_inside_cidr
tunnel1_preshared_key = "a${random_password.transfer_aws_azure_psk_1.result}" # avoid illegal 0 at beginning
tunnel1_ike_versions = ["ikev2"] tunnel2_inside_cidr = local.azure_gateway_options.gw1.tunnel2_inside_cidr
tunnel2_preshared_key = "a${random_password.transfer_aws_azure_psk_2.result}"
tunnel2_ike_versions = ["ikev2"] tags = local.azure_transfer_tags
}This connection needs to be established from Azures site, and we need to create a static route, since we don’t use bgp.
locals {
gateway_options = {
gw1-tunnel1 = {
psk = aws_vpn_connection.transfer_aws_azure.tunnel1_preshared_key
vpn_outside_ip = aws_vpn_connection.transfer_aws_azure.tunnel1_address
}
gw1-tunnel2 = {
psk = aws_vpn_connection.transfer_aws_azure.tunnel2_preshared_key
vpn_outside_ip = aws_vpn_connection.transfer_aws_azure.tunnel2_address
}
}
}resource "azurerm_local_network_gateway" "transfer_vpn_azure_aws" {
provider = azurerm.center_vpn for_each = local.gateway_options name = "transfer-vpn-azure-aws-${each.key}"
resource_group_name = azurerm_resource_group.transfer_vpn.name
location = azurerm_resource_group.transfer_vpn.location
gateway_address = each.value.vpn_outside_ip
address_space = [module.center_vpc_cidr.base_cidr_block] tags = local.azure_transfer_tags
}resource "azurerm_virtual_network_gateway_connection" "transfer_vpn_azure_aws" {
provider = azurerm.center_vpn for_each = local.gateway_options name = "transfer-vpn-azure-aws-${each.key}"
location = azurerm_resource_group.transfer_vpn.location
resource_group_name = azurerm_resource_group.transfer_vpn.name connection_protocol = "IKEv2"
type = "IPsec"
virtual_network_gateway_id = azurerm_virtual_network_gateway.transfer_vpn_azure_aws.id
local_network_gateway_id = azurerm_local_network_gateway.transfer_vpn_azure_aws[each.key].id
shared_key = each.value.psk tags = local.azure_transfer_tags
}resource "azurerm_route_table" "transfer_vpn_azure_aws" {
provider = azurerm.center_vpn
name = "aws"
location = azurerm_resource_group.transfer_vpn.location
resource_group_name = azurerm_resource_group.transfer_vpn.name route {
name = "aws"
address_prefix = module.center_vpc_cidr.base_cidr_block
next_hop_type = "VirtualNetworkGateway"
} tags = local.azure_transfer_tags
}We are left with the network configuration for AWS routing tables and the AWS VPN client.
locals {
idp_azure_group_ids = ["my-azure-group-id"]
}resource "aws_vpn_connection_route" "transfer_aws_azure" {
provider = aws.center_vpc destination_cidr_block = local.azure_cidr_range
vpn_connection_id = aws_vpn_connection.transfer_aws_azure.id
}resource "aws_route" "transfer_aws_azure" {
provider = aws.center_vpc for_each = toset(module.center_vpc.private_route_table_ids) route_table_id = each.value
destination_cidr_block = local.azure_cidr_range
gateway_id = aws_vpn_gateway.transfer_aws_azure.id
}resource "aws_ec2_client_vpn_route" "transfer_aws_azure" {
provider = aws.center_vpc for_each = toset(local.center_vpc_private_subnet_ids) client_vpn_endpoint_id = aws_ec2_client_vpn_endpoint.default.id
destination_cidr_block = local.azure_cidr_range
target_vpc_subnet_id = each.value
description = "Route to Azure CIDR range"
}resource "aws_ec2_client_vpn_authorization_rule" "transfer_aws_azure" { for_each = toset(local.idp_azure_group_ids) client_vpn_endpoint_id = aws_ec2_client_vpn_endpoint.vpn_client.id
target_network_cidr = local.azure_cidr_range
access_group_id = each.value
description = "Authorize AD group ${each.value} to connect to Azure CIDR range"
}That’s all.
Abstract
In this article, we have described how to interconnect AWS and GCP, and AWS and Azure, respectively, by means of a site-2-site VPN tunnel. For it, we have stated a HA bgp-enabled VPN configuration for AWS/GCP and a simple, i.e. two-connection, VPN configuration with static routes for AWS/Azure. Moreover, we have explained how a star-shaped hub-and-spoke VPC topology can be configured in AWS, and how a VPC client can be used to dial into thus created VPN. This client configuration has been prepared for being integrated with a network access concept based on groups provided by some IdP. The whole configuration has been presented as IaC using Terraform and HCL.
In upcoming articles, we plan to show how private Kubernetes clusters and private/public DNSes can be configured on top of this network topology.
SDA SE is a young IT company with a start-up atmosphere. In the age of digitalization SDA SE stands by its customers as a partner to jointly enable beneficial value-added services, which improve the communication and interaction with their end customers. The customer’s competitiveness is increased by creating a world-class digital consumer experience. Our platform and modular product building blocks are the basis for combining existing resources and solutions with third parties in the market. We are supported in our venture by our investors and partners, such as .msg, Allianz X, Debeka, HUK Coburg and Signal Iduna.
You have the chance to become part of our rapidly growing company. Thus, you have the opportunity to take on responsibility at an early stage and actively shape our further development. You will be supported by 50 dedicated colleagues and state-of-the-art technologies. We live an appreciative company culture, which leaves room for humor and individuality.
Click here to see our open positions. Apply now and join our team!
