AWS, Ubuntu 14, and Multiple ENI’s

I used to run Linux as my desktop, and when I did, I was all in. I ran Gentoo Linux, who’s claim to fame is that it compiles EVERYTHING — no packages. I really enjoyed tinkering with every aspect of the OS and learning things the hard way, pretty much daily.

Then I got busy, and then I go busier still. I didn’t have time left in my day to tinker with my workstation anymore. So I switched to Mac’s and OS X, as they gave access to Linux like tools, on a BSD underpinning, with a “get things done everytime” user interface. I’m still Linux oriented, but these days I mostly live in AWS Linux. That said, I still get a healthy exposure to Ubuntu, RHEL/CentOS, and the occasional SUSE.

Getting to the point, I’m spoiled. When I add an Elastic Network Interface (“ENI”) to an AWS Linux instance *POP* up it comes. Not so much the case in Ubuntu. As I tinkered with this (in relation to my earlier ENI post) I took some more recent notes on how to configure an Ubuntu Box to support a second ENI, and thought I’d share.

First of all, you have to get into the box you are adding the interface to, so modify the Security Groups, and/or routing to get into the primary interface via SSH. Having a “Bastian” or “Jump” box is a handy thing here as well.

After that, you will need to add an ENI via the CLI or the AWS Management Console, and attach it to your instance. Now that the AWS side is ready to go, lets start tweaking the OS to accept it.

NOTE: You will need to SUDO SU to root, or just SUDO to execute the commands needed to bring up the interface.

Change directories to the /etc/network/interfaces.d directory. You will need to copy the eth0.cfg file and rename it to eth1.cfg.

This file initializes the NIC’s on the OS, which are in reality the ENI’s that you attached to the instance. Eth0 is automatically supported, but eth1 will need some tweaking — so open up the file with your favorite editor (May I suggest Nano?) and make it look as such:

You will basically take all the “eth0" entries and change them to “eth1”.

Save the file, and get back out to the command line — from here we will bring up the interface, and capture some information from it to finalize the interface configuration. Run the “ifup eth1” command to bring up the interface.

This command brings up the interface, and based on the config file we modified, the system reached out to a DHCP server and asked for an IP address on the subnet that the ENI is attached to. IP’s assigned to ENI’s typically do not change in AWS after they are assigned by the platform. To finish up our config, we need to have the IP address of the DHCP “server” that AWS is using on that subnet, so we can refrence it in our commands. Look at the feed back of your command — the line that starts with “DHCPOFFER” indicates that an address was offered by — that’s our DHCP origin address. Write that down.

This next step is optional, but I like to see that everything is coming together. Run “ifconfig” to verify that the interface is up

Sweet! Next we need to get some routes in the system so that traffic can flow into and out of the new interface we created. First, we are going to pick up the default route advertised by the DHCP server to our system.

The address is the DHCP server IP you wrote down earlier.

Then, we will add routes to to the internal table to support proper connectivity, we will need a “from” and a “to” route.

The address is the IP assigned to the ETH1 interface from the DHCP server.

Now if you hooked up an ENI with an Elastic IP (“EIP”), made sure that the routing tables were correct, and allow ICMP Ping through the Security Group that the ENI belongs to (Phew! That’s a lot of TLAs! -> Three Letter Acronyms) you should be able to ping the EIP assigned to the ENI, attached to the instance!

IF this doesn’t work, make sure that the AWS side of things is set up correctly. 90% of the time, that’s the problem is routing or Security Groups — look there first.

Now that we have hooked this up by hand, let’s make it persistant between boots. We will add the commnands that we just ran to the eth1.cfg file, and interestingly enough, will need to add a couple additional commands to make sure that the new interface plays well with others on boot. Modify your eth1.cfg file to look like this — with your specific information of course.

A few notes on this config:

  1. Notice that the commands have and “up” in front of them. This commands the system to wait until the interface is in an up state before executing the command — this is a critical addition.
  2. The last line, “ip route flush cache”, does what is says — flushes the route cache. If you don’t add this line along with the others, you get an odd behavior where eth0 stops responding becauase of route cache confusion, and you can only access eth1.

Reboot the Instance, either from the OS, or from the AWS Console. It should come up and have two interfaces enabled and ready to rock on two different subnets.

This knowledge is handy for adding an interface on a second subnet, but just as an aside, adding two interfaces to the same subnet on the same instance in AWS can introduce some “intresting” routing behaviours to the intstance. YMMV.

I’m not 100% sure that these instructions will work with pure Debian systems, but it should be close. Only trick here is that “close” in Linux is sometimes miles apart.

On that note, there is a way to attempt to give Ubuntu the same sort of automatic ENI addition functionality of AWS Linux, it’s called ubuntu-ec2net. The only problem is that it hasn’t really been updated for 3 yrs, and is a bit twitchy — User beware.