How to drop 10 million packets per second

by Marek Majkowski

Internally our DDoS mitigation team is sometimes called “the packet droppers”. When other teams build exciting products to do smart things with the traffic that passes through our network, we take joy in discovering novel ways of discarding it.

CC BY-SA 2.0 image by Brian Evans

Being able to quickly discard packets is very important to withstand DDoS attacks.

Dropping packets hitting our servers, as simple as it sounds, can be done on multiple layers. Each technique has its advantages and limitations. In this blog post we’ll review all the techniques we tried thus far.

Test bench

To illustrate the relative performance of the methods we’ll show some numbers. The benchmarks are synthetic, so take the numbers with a grain of salt. We’ll use one of our Intel servers, with a 10Gbps network card. The hardware details aren’t too important, since the tests are prepared to show the operating system, not hardware, limitations.

Our testing setup is prepared as follows:

• We transmit a large number of tiny UDP packets, reaching 14Mpps (millions packets per second).
• This traffic is directed towards a single CPU on a target server.
• We measure the number of packets handled by the kernel on that one CPU.

We’re not trying to maximize userspace application speed, nor packet throughput — instead, we’re trying to specifically show kernel bottlenecks.

The synthetic traffic is prepared to put maximum stress on conntrack — it uses random source IP and port fields. Tcpdump will show it like this:

$ tcpdump -ni vlan100 -c 10 -t udp and dst port 1234 
IP 198.18.40.55.32059 > 198.18.0.12.1234: UDP, length 16 
IP 198.18.51.16.30852 > 198.18.0.12.1234: UDP, length 16 
IP 198.18.35.51.61823 > 198.18.0.12.1234: UDP, length 16 
IP 198.18.44.42.30344 > 198.18.0.12.1234: UDP, length 16 
IP 198.18.106.227.38592 > 198.18.0.12.1234: UDP, length 16 
IP 198.18.48.67.19533 > 198.18.0.12.1234: UDP, length 16 
IP 198.18.49.38.40566 > 198.18.0.12.1234: UDP, length 16 
IP 198.18.50.73.22989 > 198.18.0.12.1234: UDP, length 16 
IP 198.18.43.204.37895 > 198.18.0.12.1234: UDP, length 16 
IP 198.18.104.128.1543 > 198.18.0.12.1234: UDP, length 1

On the target side all of the packets are going to be forwarded to exactly one RX queue, therefore one CPU. We do this with hardware flow steering:

ethtool -N ext0 flow-type udp4 dst-ip 198.18.0.12 dst-port 1234 action 2

To review the complete code snippets and the rest of this post please visit the original on the Cloudflare blog.

Originally published at blog.cloudflare.com on July 6, 2018.