How to setup a site to site VPN and networking with AWS and IBM Cloud
This is part of the “Great migration” blog post series describing how we migrated from IBM Cloud to AWS.
One of the first things we had to do when migrating from IBM Cloud to AWS was to work out the networking connectivity between IBM Cloud and AWS as we wanted to guarantee user data would be safe during the overall migration process. We want to ensure:
- Workloads in AWS will be able to communicate with workloads in IBM Cloud, and vice versa.
- Communications be done over a private and encrypted channel.
- Our application remained performant and that latency or bandwidth were not impacted by cross cloud communication.
First start performance benchmarking
To test bandwidth and latency between AWS and IBM, we setup a simple ping + SFTP test between IBM and AWS. We did this by spinning up an EC2 instance in AWS and have it connect to an existing server in IBM.
One of the “weird” things about IBM Cloud Classic infrastructure is that all servers in IBM are given a public IP address. You don’t have much of a choice in this, but in this scenario it made things pretty easy for us. We created a simple script to ping from the EC2 server to the existing server in IBM and in the other direction. In our case, the regions in IBM and AWS were in the same city and we found the ping to be under 10ms, usually hovering around the 5ms. First test done... ping over the internet between the cloud providers was consistently under 10ms.
With latency not a major issue for us, we then SFTP 10GB files in both directions over the public IP addresses to measure the bandwidth. The file transfers were constantly hovering around 100MB/s. We compared this to the amount of traffic and speed we saw internally on our main database, and saw that the database traffic below 100MB/s, usually around the 90MB/s. Based on this, we made the assumption that the public internet bandwidth between IBM and AWS was acceptable.
Once we had this information, we explored the idea of setting a “Direct Connect”, however this was quite costly and beyond our budget. Thus, we decided that a site-to-site VPN between the 2 cloud providers would fulfil our requirements whilst remaining cost effective.
Pick your IP range in AWS and setup the VPC
IBM Cloud Classic Infrastructure give their customers a predefined VLAN slice which has a dedicated subnet in the 10.x.x.x/8 IP range. You can find the defined range of IP addresses based on the region from IBM here. Once we knew the IP address in IBM, we had to pick a 10.x.x.x/16 IP range in AWS for our VPCs that didn’t overlap with the IP range for our region in IBM. This VPC will be used to host the new servers and services in AWS which we plan to migrate from IBM. We used a predefined terraform module to set this up in AWS.
Here’s a code snippet for our VPC setup (note these are not real values):
vpc.tf:
module "vpc" {
source = "modules/vpc"
sws_team = var.sws_team
sws_env = var.sws_envvpc_name = "vpc-${var.sws_env}"
cidr = "10.3.0.0/16"
azs = ["us-west-1a", "us-west-1b", "us-west-1c"]
database_subnets = ["10.3.32.0/24", "10.3.33.0/24", "10.3.34.0/24"]
private_subnets = ["10.3.0.0/24", "10.3.8.0/21", "10.3.16.0/21"]
public_subnets = ["10.3.101.0/24", "10.3.102.0/24", "10.3.103.0/24"]
intra_subnets = ["10.3.51.0/24", "10.3.52.0/24", "10.3.53.0/24"]
}
modules/vpc/main.tf
module "vpc" {source = "terraform-aws-modules/vpc/aws"
version = "3.0.0"name = var.vpc_name
cidr = var.cidrazs = var.azs
database_subnets = var.database_subnets
private_subnets = var.private_subnets
private_subnet_tags = {
Tier = "Private"
}public_subnets = var.public_subnets
public_subnet_tags = {
Tier = "Public"
}
intra_subnets = var.intra_subnetsenable_nat_gateway = true
single_nat_gateway = false
one_nat_gateway_per_az = true
enable_dns_hostnames = true
reuse_nat_ips = true
external_nat_ip_ids = "${aws_eip.nat.*.id}"tags = {
Name = var.vpc_name
}
}resource "aws_eip" "nat" {
count = 3
vpc = true
}
Set up the site to site VPN
Whilst we have a code first methodology at Simply Wall St, one of the first decisions we made was to not put the site-to-site VPN configuration in code. The main reason was that configuring VPNs are notoriously fiddly, as well as the fact that this was an inherently temporary piece of infrastructure. There are a lot of settings to play around with to get VPNs working, so we decided to set this up manually by hand.
To setup a site to site VPN with IBM and AWS, you will need VPN devices on both sides. On the IBM side, you are given the options of either a Vyatta or Juniper gateway device for setting up a site-to-site VPN. Both of which require a lot of manual setup. We reached out to IBM support, and they recommended the Juniper Gateway device. Whilst more expensive than Vyatta’s, we decided that it was safer to implement the vendor’s recommendation.
At a high level, this was our network architecture:
On the left hand-side, we have IBM Cloud very roughly represented in two data centres called WDC01 and WDC02. Each data centre in IBM has its own set of highly available Juniper Gateway devices which would serve as a VPN between AWS and IBM. On the AWS side, 2 site-to-site VPNs were created in the target VPC. Provisioning these is pretty easy on both sides. We followed IBM’s Juniper instructions to get the basics setup, and AWS’s instructions on how to setup site to site VPN. At this point, we did not have both of them talking to each other just yet; this proved to be quite difficult.
When you login to the AWS’s site to site VPN console, you are able to download a predefined set of instructions to connect to Juniper gateways.
When applying this suggested settings given to the Juniper gateway devices in IBM, we found that the tunnel was not stable. We observed that the tunnels would come up, and when attempting to send a large amount of data across using SFTP as a sanity check, the transfer would “pause” half way through for few seconds and then resume. We raised a case with IBM but they were not much help trying to get us to debug. We raised the case with AWS, and they were able to point out that the Juniper Gateways was sending deletes for Phase 2 negotiation every 2 minutes which was bringing down the tunnel on a regular basis. We were able to find the corresponding errors on the Juniper gateway KMD logs:
[Aug 26 22:58:20]ikev2_packet_st_send: FSM_SET_NEXT:ikev2_packet_st_send_done
[Aug 26 22:58:21]Deleted (spi=0x81a62800, protocol=ESP dst=x.x.x.x) entry from the peer hash table. Reason: VPN monitoring detected tunnel as down. Existing IPSec SAs cleared
[Aug 26 22:58:21]In iked_ipsec_sa_pair_delete Deleting GENCFG msg with key; Tunnel = 131074;SPI-In = 0x81a62800
[Aug 26 22:58:21]Deleted SA pair for tunnel = 131074 with SPI-In = 0x81a62800 to kernel
[Aug 26 22:58:21]Deleted (spi=0x81a62800, protocol=ESP) entry from the inbound sa spi hash table
[Aug 26 22:58:21]Deleted (spi=0xc15ea3bb, protocol=ESP dst=y.y.y.y) entry from the peer hash table. Reason: VPN monitoring detected tunnel as down. Existing IPSec SAs cleared
[Aug 26 22:58:21]NHTB entry not found. Not deleting NHTB entry
[Aug 26 22:58:21]NHTB entry not found. Not deleting NHTB entry
[Aug 26 22:58:21]NHTB entry not found. Not deleting NHTB entry
[Aug 26 22:58:21]ikev2_state_info_initiator_out: FSM_SET_NEXT:ikev2_state_info_initiator_out_add_delete
[Aug 26 22:58:21]ikev2_state_info_initiator_out_add_delete: FSM_SET_NEXT:ikev2_state_info_initiator_out_add_notify
[Aug 26 22:58:21]ikev2_state_info_initiator_out_add_notify: FSM_SET_NEXT:ikev2_state_info_initiator_out_add_conf
[Aug 26 22:58:21]ikev2_state_info_initiator_out_add_conf: FSM_SET_NEXT:ikev2_state_notify_vid_encrypt_send
[Aug 26 22:58:21]ikev2_state_notify_vid_encrypt_send: FSM_SET_NEXT:ikev2_state_notify
[Aug 26 22:58:21]ikev2_state_notify: FSM_SET_NEXT:ikev2_state_vid
[Aug 26 22:58:21]ikev2_state_vid: FSM_SET_NEXT:ikev2_state_private_payload
[Aug 26 22:58:21]ikev2_state_private_payload: FSM_SET_NEXT:ikev2_state_encrypt
[Aug 26 22:58:21]ikev2_state_encrypt: FSM_SET_NEXT:ikev2_state_send
[Aug 26 22:58:21]ikev2_list_packet_payloads: Sending packet: HDR, DEL
[Aug 26 22:58:21]IKEv2 packet S(<none>:500 -> z.z.z.z:500): len= 69, mID=183, HDR, DEL
[Aug 26 22:58:21]ikev2_packet_st_send_request_address: FSM_SET_NEXT:ikev2_packet_st_send
[Aug 26 22:58:21]ikev2_udp_send_packet: [9513400/95d4700] <-------- Sending packet - length = 0 VR id 0The VPN monitoring given in the default Juniper configuration provided from AWS was terminating the tunnel regularly, and by removing the VPN monitor we resolved the issue. We also had to combine the best practices from Juniper with the AWS default configuration.
If you are interested in our configuration, this is snippet of our configuration for our Juniper gateway which has the combined recommended best practices from IBM + configuration from AWS — health checks to get it all working:
security {
ike {
traceoptions {
file kmd size 1024768 files 10;
flag all;
}
proposal ike-prop-aws-vpn-1 {
authentication-method pre-shared-keys;
dh-group group2;
encryption-algorithm aes-128-gcm;
lifetime-seconds 28800;
}
proposal ike-prop-aws-vpn-2 {
authentication-method pre-shared-keys;
dh-group group2;
encryption-algorithm aes-128-gcm;
lifetime-seconds 28800;
}
policy ike-pol-aws-vpn-1 {
mode main;
proposals ike-prop-aws-vpn-1;
pre-shared-key ascii-text "1111"; ## SECRET-DATA
}
policy ike-pol-aws-vpn-2 {
mode main;
proposals ike-prop-aws-vpn-2;
pre-shared-key ascii-text "2222"; ## SECRET-DATA
}
gateway gw-aws-vpn-1 {
ike-policy ike-pol-aws-vpn-1;
address 1.1.1.1; #AWS's public IP for tunnel 1
dead-peer-detection {
interval 10;
threshold 3;
}
no-nat-traversal;
nat-keepalive 3;
external-interface reth1.0;
version v2-only;
}
gateway gw-aws-vpn-2 {
ike-policy ike-pol-aws-vpn-2;
address 2.2.2.2; #AWS's public IP for tunnel 2
dead-peer-detection {
interval 10;
threshold 3;
}
no-nat-traversal;
nat-keepalive 3;
external-interface reth1.0;
version v2-only;
}
}
ipsec {
proposal ipsec-prop-aws-vpn-1 {
protocol esp;
encryption-algorithm aes-128-gcm;
lifetime-seconds 3600;
}
proposal ipsec-prop-aws-vpn-2 {
protocol esp;
encryption-algorithm aes-128-gcm;
lifetime-seconds 3600;
}
policy ipsec-pol-aws-vpn-1 {
perfect-forward-secrecy {
keys group2;
}
proposals ipsec-prop-aws-vpn-1;
}
policy ipsec-pol-aws-vpn-2 {
perfect-forward-secrecy {
keys group2;
}
proposals ipsec-prop-aws-vpn-2;
}
vpn aws-vpn-1 {
bind-interface st0.10;
df-bit clear;
ike {
gateway gw-aws-vpn-1;
ipsec-policy ipsec-pol-aws-vpn-1;
}
establish-tunnels immediately;
}
vpn aws-vpn-2 {
bind-interface st0.20;
df-bit clear;
ike {
gateway gw-aws-vpn-2;
ipsec-policy ipsec-pol-aws-vpn-2;
}
establish-tunnels immediately;
}
}
}Set up network route tables
With the VPN tunnel setup, we need to test the following scenarios:
- Servers in IBM can communicate with servers in AWS
- Servers in AWS can communicate with servers in IBM on the new VLAN
- Servers in AWS can communicate with databases in IBM on the old VLAN
Below is a diagram depicting the subnets assigned to each Cloud/VLAN. IP Address ranges are generalised for privacy reasons, but it should be enough to understand some of the problems we faced and how we solved it.
On the IBM Side, IBM essentially controls the entire 10.0.0.0/8 IP space on their network devices. You are not able to control the default route tables on the default IBM gateway devices. The “VLAN OLD” has our databases running on them and we didn’t want to move them to the new VLANs as we considered this too risky. New Kubernetes nodes were now provisioned on WDC01 VLAN NEW & WDC02 VLAN NEW. Because these VLANS are managed by the Juniper Gateways, we have full control on the routing and security configuration on the Juniper Gateway devices. On the AWS side, we had a VPC whose IP space (10.3.0.0/16) was a subset of the 10.0.0.0/8.
Setting up routing for AWS VPC to/from WDC01- VLAN NEW & WDC02 — VLAN NEW was relatively easy. On the AWS side, the route tables looked like:
On both IBM Juniper gateway devices, we had this route table:
With the setup above, servers in AWS and VLAN NEW on WDC01 and WDC02 are able to route nicely.
Setting up routing to/from VLAN NEW on WDC01 and WDC02 to WDC02 VLAN OLD was also relatively straight forward.
On the Juniper Gateways
And because IBM knows that 10.1.0.0/16 and 10.2.0.0/16 are on WDC01 VLAN NEW and WDC02 VLAN NEW respectively, the default IBM Gateway device knows how to route traffic to the destinations.
Setting up routing from AWS to/from WDC02 VLAN OLD was more difficult. Because you don’t get control of the Default IBM Gateway Device, we couldn’t control routing to/from the AWS IP range 10.3.0.0/16 as it thinks that IBM owns the entire 10.0.0.0/8 IP range. What we ended up doing was a Source NAT on the Juniper Gateway Device in WDC02. The Source NAT table was setup so that traffic from AWS would appear to get an IP from the 10.2.0.0/16 range. This means that once traffic hits the Default IBM Gateway Device, it would appear as if it came from the WDC02 VLAN NEW; so the Default IBM Gateway Device would know how to route the IP packets back. The Source NAT tables are maintained on the Juniper Gateway Device in WDC02; and thus knows to translate the destination packet back to AWS’s VPN. There are a few different ways to configure Source NAT on Juniper gateways, we chose to assign an IP from a pool of 100s in the 10.2.0.0/16 range for each source IP in AWS for performance reasons and we knew we wouldn’t have more than 100 addresses in AWS as part of the migration.
For this to work, we needed to add an extra route on AWS to ensure traffic from AWS to VLAN-OLD is routable on the AWS side.
This solved the problem for traffic from AWS to VLAN-OLD; but it didn’t solve traffic originating from VLAN-OLD to AWS. We knew this was going to be a problem and thus we moved all compute except databases on VLAN OLD to VLAN NEW. The reason being, databases generally don’t initiate connections, the generally receive connections. This was good enough compromise for our migration.
nat {
source {
pool src-nat-pool-1 {
address {
10.2.0.100/32 to 10.2.0.200/32;
}
}
session-persistence-scan;
session-drop-hold-down 28800;
port-randomization disable;
rule-set aws-to-ciq {
from zone AWS-PRIVATE;
to zone SL-PRIVATE;
rule r1 {
match {
source-address 10.3.0.0/16;
destination-address [ 10.0.0.0/16 ];
}
then {
source-nat {
pool {
src-nat-pool-1;
}
}
}
}
}
}
}Transfer all workloads in IBM to use the new Juniper gateway device
With the VPN setup, it was now time to switch all workloads in IBM to use the Juniper gateway device. Unfortunately in IBM world, ALL private workloads must be switched to use the Juniper gateway device and you can no longer use the default IBM devices for traffic once you attach a VLAN to the Juniper gateways and turn it on.
To avoid a big bang approach of switching all private traffic to Juniper, we provisioned new private VLANS in IBM and attached it to the Juniper gateways. We then created new Kubernetes worker pools attached to the newly provisioned VLAN that’s attached to the Juniper gateway. These new worker pools only had 1 server in each data centre to ensure not not many pods were scheduled on these new nodes. Once we were confident that the new nodes worked, we slowly moved all nodes to the new worker nodes and decommissioned the old worker nodes.
We left our databases on the existing VLANs, but enabled VLANs spanning to ensure the databases were reachable from pods on the new VLAN.
Here’s a snippet of our Juniper configuration describing the VLAN setup on WDC02. You can see we have VLAN-123 attached to the reth2 interface.
interfaces {
... existing interfaces ...
reth2 {
vlan-tagging;
redundant-ether-options {
redundancy-group 1;
}
unit 123 {
vlan-id 123;
family inet {
address 10.2.0.1/16;
}
}
}
... other interfaces ...
}Configure IBM Network bandwidth pooling to save money
In the setup above, all traffic goes through the public internet on the site-to-site VPN meaning the egress side can get very expensive if you’re sending 100’s terabytes of data across each month. Both IBM and AWS charge for egress traffic. There’s not a lot you can do on the AWS side, but on the IBM side, you are able to utilise “Network Bandwidth Pools”; by provisioning bare-metal servers with 20TB of public egress bandwidth each month, you can combine the entire fleet of bare-metal servers’ public egress so the total is calculated, and not charged on a per server basis. The cheapest bare-metal server in IBM is around $250 per month, if you were to pay for 20TB of on demand network costs, it’ll cost around $1000 if you’re in US/Canada/Europe. So it makes sense to provision “dummy” bare metal servers to save on public egress cost on the IBM side.
Reference configuration
See sanitised version of our entire Juniper Gateway configuration which ties all the concepts described above for Juniper in WDC02.
