So Amazon Develops Its Own NIC — AWS Enhanced Networking!

Ethernet adapter types available for EC2

AWS EC2 instances offer 3 types of network adapters (depending upon the instance types) — the default VIF, which offers low (~100Mbps) to moderate (~300Mbps) network throughput, and two other network adapters that support enhanced networking (~10Gbps or greater) — Intel 82599 Virtual Function adapter, and the next generation Elastic Network Adapter (ENA). Enhanced networking also requires specialized H/W support (the above mentioned physical network adapters installed on the EC2 instance host) and thus this feature is only available through specific EC2 instance types.

Enhanced Networking

The VIF adapter present in an EC2 instance is provided by the underlying virtualization layer i.e. XEN hypervisor for AWS. This adapter employs usual network virtualization technique that involves significant overheads as it relies on traditional interrupt based (IRQ) approach inherent within the PCIe NIC design.

In a attempt to support higher network throughput, Amazon Web Services introduced enhanced networking for which they relied on PCIe NICs equipped with SR-IOV technology that allows a VM (EC2 instance) to bypass the underlying hypervisor (XEN) and use direct memory access (DMA) instead of interrupting the CPU (more about SR-IOV in the last section of this article).

Initially, Amazon offered 10Gbps network throughput for a few EC2 instance types using Intel’s 82599 VF PCIe NIC. In January 2015, Amazon bought an Israel based chip maker Annapurna Labs; based upon the newly acquired company’s flagship product Alpine, Amazon launched its own new generation PCIe NIC that supported upto 25Gbps of network throughput. Amazon christened this NIC as Elastic Network Adapter (ENA). ENA is only available for some specific EC2 instance types.

The following table, based on a similar table from Amazon Knowledge Center page on Enhanced Networking, summarizes the network adapter types and which EC2 instances types they are available for as of today -

How do I enable enhanced networking on EC2 instances

Refer Amazon documentation on enabling Enhanced Networking for EC2 instances.

The technology behind Enhanced Networking — SR-IOV

Assigning a dedicated PCI Network H/W adapter (or port) for each VM on a host can give a line rate throughput but it is not feasible, and software based sharing of IO device (IO Virtualization) imposes significant overheads and thus unable to use the capabilities of the physical device fully.

Single Root — Input Output Virtualization (SR-IOV) specification, released by PCI SIGin 2007, is one of the technologies to achieve Network Function Virtualization (NFV). It gets its name “Single Root” from the PCIe Root Complex. SR-IOV enables the physical network adapter to be shared directly with the VMs bypassing the hypervisor as shown below.

SR-IOV architecture offers two function types -

  • Physical Functions (PFs) — A NIC feature for configuring and managing SR-IOV functionality on the deivce. It is exploited by PF device driver that is part of the hypervisor.
  • Virtual Functions (VFs) — It is a PCIe function that is used by the respective VMs for communicating directly with the physical NIC. The hypervisor, using the PF device driver, assigns VFs to VMs and then VFs use native virtual function device drivers to directly communicate with the NIC.

So when a data packet is received by the NIC adapter, the classifier at SR-IOV capable NIC — as configured by the PF device driver — places it to the appropriate queue mapped to the appropriate virtual function and its target VM.

Question to ponder upon …

Since SR-IOV lets a VM directly map to a PCI port bypassing the hypervisor, how does Amazon achieve Inter-EC2 network switching, implement Security Groups or ACLs?

Hint.

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

Resources:

Amazon’s docs, blogs & videos:

  1. Read #9 “The importance of the network” @ All Things Distributed
  2. How do I enable and configure enhanced networking on my EC2 instances?
  3. Enhanced Networking — FAQs
  4. AWS re:Invent 2016: Optimizing Network Performance for Amazon EC2 Instances
  5. AWS re:Invent 2016: Optimizing Network Performance @ YouTube
  6. AWS re:Invent 2016: James Hamilton @ YouTube (between 23:00–36:00 minutes)
  7. Elastic Network Adapter — High Performance Network Interface for Amazon EC2
  8. Amazon EC2 Instance Types

Independent blogs & videos:

  1. How did they build that — EC2 Enhanced Networking?
  2. Single Root I/O Virtualization (SR-IOV) Primer @ RedHat
  3. An Introduction to SR-IOV Technology @ Intel
  4. Single Root I/O Virtualization (SR-IOV) @ VMWare
  5. Accelerating the NFV Data Plane: SR-IOV and DPDK… in my own words
  6. Red Hat Enterprise Linux OpenStack Platform 6: SR-IOV Networking — Part I: Understanding the Basics
  7. Kernel bypass
  8. Network function virtualization (NFV)
  9. PCI Express Architecture In a Nutshell
  10. Amazon buys secretive chip maker Annapurna Labs for $350 million
  11. The chip company Amazon bought for $350 million has a new product that could terrify Intel
  12. Annapurna Labs
  13. Intel VMDq Explanation by Patrick Kutch @ YouTube
  14. Intel SR-IOV Explanation by Patrick Kutch @ YouTube
  15. What is SR-IOV?