AWS Networking — ENIs Through the fog of Jet Lag
Jet lag is the mind killer.
Anyone who has ever had to regularly brave the US Airport circuit to make sure the checks don’t bounce at the end of the month, understands that jetlag isn’t something that only gets you the day after you travel. The confused, fuzzy headed filter will occasionally slam down on your poor hapless brain anywhere from 24 to 168 hours since you finally landed back in your home time zone.
Delayed jet lag hit me like a sack of bricks yesterday, as I tried to explain verbally to a Client how to attach an Elastic Network Interface (ENI) to a network isolated AWS instance to support remote administration. While I was fighting my eye lids for control of my consciousness, I didn’t have the luxury of muscle memory to demonstrate the concept, since I wasn’t driving the console. To top it off, we were also working on a wireless connection that seemed to be playing merry hell with our AWS Console session.
Needless to say, hilarity ensued as Murphy played with us like marionettes. At the end of the session we did get the connection up, and I was reminded that AWS, in an attempt to keep things simple, obscures what is really going on when dealing with network interfaces, Elastic IP’s and routing tables inside of a Virtual Private Cloud (VPC). My understanding was complete, but in my fog I assumed that everyone understood the subtleties that are hidden in plain site when working with the console. That led to a pretty disjointed Point-A to Point-B process. Ultimatly it demonstrated a number of little things that the team hadn’t seen from the perspective of the ENI.
As such, and in an attempt to transform my foggy-struggles at the end of a simple consulting session into a teachable moment, I’d like to give you another way to look at AWS networking that isn’t new, but is often obscured by the inherit assumptions in many AWS diagrams.
Simplified, yes — however this is typically how AWS diagrams show the relationships of instances, subnets, the VPC, and Security groups. I often drill down deeper, but many examples and descriptions show that the instance has a direct membership INSIDE of a subnet, and that the Internet Gateway (IGW) is lurking near the public subnets waiting to route traffic with baited breath. In reality, its not that way at all.
The way we really virtually wire all the instances together in a VPC is much more like the diagram above. It’s not as pretty and symmetrical, but conceptually you should always be thinking about your VPC networking like this. When you do, the security, flexibility and functionality of the VPC networking model spring forward.
First of all, ENIs are attached to instances and assigned to subnets. An instance’s connection, or anchor, to subnets is through the ENIs, or if you prefer, hook into a subnet trunk, though the ENI. Otherwise they float on the edge of subnets. The thing that makes the generalized “instance in a subnet” diagram technically correct, is that each instance is given a non-movable ETH0 interface at launch time that is assigned to a subnet. Really it’s the difference between looking at a visualization as an application/systems engineer or a network engineer. Using AWS requires you to do both in order to get the full picture.
The VPC router links all the subnets within a VPC together, referencing route tables that subnets are associated with. The router makes the call on what paths the traffic will take between subnets, and to ingress/egress points like the IGW or Virtual Gateways (VGWs) that you have attached to the VPC. Finally, the security groups encapsulate the ENI’s — not the instances — so that the Security Groups, ENI’s, and subnets act as membranes to protect and service the instances.
In this context, it is also easier to show/visualize the true nature of Security Group memberships. Since traditional firewalling typically defended a subnet, sometimes the mind will drift back to that assumption. Looking at the VPC from an ENI centric view changes that for the better.
So, drilling into the use case I was trying to explain to my Client earlier, we have a need to preform maintenance or house keeping functions on our DB server, however it is in a Private Subnet. We can take advantage of the magic of ENIs and Elastic IPs (EIPs) to get the job done securely when we don’t have the luxury of using a Virtual Private Network (VPN) connection to come in over a permanent established backchannel.
The first part of this adventure is to build an ENI to handle the traffic. We need to do this because right now the DB server is only connected to the Private Subnet A, and Private Subnet A uses the NAT instance to communicate out of the VPC to the Internet. That is only a one-way path leading out of the VPC, that can’t support incoming SSH sessions into the DB instance, or interactive tools DB tools like Navicat.
When we build the ENI, we specify the Subnet that the ENI is attached to, and what Security Group it is attached to as well.
In the AWS Architecture and SysOps classes I teach, I stress that there are two places to look first when you are having a connectivity issue: Security Groups and Routing Tables. 90% of the time, there is a SNAFU with one of those two configuration points. When attaching your ENI to a Security Group to support this use case, make sure you have configured the group to support the bare necessity of ports, and make absolutely sure you have specified a source IP address or source IP range to allow the traffic from. While SSH can weather unfiltered exposure fairly well if you are using Key authentication, a system with RDP open to the world will quickly find itself a new occupation as a Bit-Torrent node, compliments of some pimply faced 133t script kiddie.
Specifically, we will need to create a Security Group for our use case, let’s call it “MySQL-Direct-Access”, and open up ports tcp/22 and tcp/3306 from an exact IP address used by our SQL team. We will assign the ENI to that group, and to the Public A Subnet.
After that, we will attach the ENI to the DB server, and associate an EIP with the ENI.
(Note: The AWS Linux OS picks up the new ENI automatically, and oddly enough *most* Windows AMI’s do as well. However Ubuntu, SUSE, and RHEL sometimes don’t and you’ll need to modify the standard config files to get the instance to see the new ENI. Sometimes a restart will do the trick too)
That changes our logical landscape to look like this
The routing table attached to the Public Subnet will route traffic to and from our new ENI attached to our DB server. It will show up as ETH1 on the instance and in the AWS Console / API. The Security Group will allow ports tcp/22 and tcp/3306 to be accessed by a specific system or IP grouping of systems. The EIP atached to the ENI will be the target for our remote administrators. When there is no need for the interface to be active, we can disattach it from the DB instance. We could also give the DB or SA teams specific permissions in the AWS platform that would let them attach it when needed, or use a script to bring it online.
This is a pretty simple use case, for a condition that comes up more often than you’d think. I’d like the stress that the best way to handle this is to have a Direct Connect (DX) or a Virtual Private Network (VPN) into the VPC to support privileged and/or trusted connections. That said, every situation and customer is a unique snowflake, and this “Management ENI” technique is infinitely better than just tossing your DB servers into a public subnet.
Now, if you all don’t mind, I’m going to go find another cup of coffee to keep away Jet lag Gremlins.