Capturing HTTP packets the hard way

Over the years I’ve used tcpdump multiple times to inspect the network traffic coming and going from my machine, but not once did I bother asking myself how it worked. I naïvely assumed tcpdump had some way of eavesdropping on all the packets going through my network interface and would filter out the ones I asked for and output them.

Last week I attended a Papers We Love meetup for the first time. The topic: “The BSD Packet Filter: A New Architecture for User-level Packet Capture”. Little did I know I would leave with a new appreciation for tcpdump.

As it turns out, copying all these packets into user space in order for tcpdump to filter them would be expensive and slow.

Enter BPF (Berkeley Packet Filter).

BPF is a virtual machine that lives within the kernel and its job is to filter packets.

When you run tcpdump -i en0 "ip and tcp", the string "ip and tcp" gets compiled by pcap_compile into a BPF program. The syntax for this input string is specified in pcap-filter.

The generated BPF program for the filter "ip and tcp" looks like this:

$ tcpdump -i en0 -d "ip and tcp"
(000) ldh [12]
(001) jeq #0x800 jt 2 jf 5
(002) ldb [23]
(003) jeq #0x6 jt 4 jf 5
(004) ret #262144
(005) ret #0

The kernel will then use this BPF program filter all packets that go through the en0 device.

Since we’re dealing with an ethernet device, the packet filtered by the BPF program will be an ethernet packet. You can think of this packet as simply an array of bytes.

Here’s an excerpt of an ethernet packet that contains the beginning of an HTTP request to example.com:

Let’s analyze the program instruction by instruction.

(000) ldh [12]
(001) jeq #0x800 jt 2 jf 5

ldh [12] means load a half-word (2 bytes) starting at the offset 12 of the packet and store it in register A.

jeq #0x800 jt 2 jf 5 will compare the value in register A with the hex number 0x800. If they are equal, the program will jump to instruction 2 otherwise it will jump to instruction 5.

The offset 12 and the value 0x800 were a bit of a mystery to me until I discovered this image:

What these two instructions are actually doing is checking if the EtherType is equal to 0x800 which means the frame contains an IPv4 datagram.

We can see this is the case for our example packet:

The next two instructions are similar.

(002) ldb [23]
(003) jeq #0x6 jt 4 jf 5

ldb [23] will load a byte at offset 23 of the packet and store it in register A.

jeq #0x6 jt 4 jf 5 will compare the hex value 0x6 with the value in register A.

Again, the values 23 and 0x6 were a bit of a mystery to me until I discovered the IPv4 header format:

According to the table above, the 9th byte in the IPv4 header is the protocol.

Wait a minute… 🤔 If the protocol is the 9th byte, why is the BPF program reading the 23rd byte?

The IPv4 packet is contained within the ethernet frame. The ethernet frame MAC header is 14 bytes long, so we need to add that to the offset 9 which gives us 23.

So what these two instructions are actually doing is checking if the protocol of the IPv4 packet is equal to 0x6 which means TCP.

Finally, the last two instructions are return statements.

(004) ret #262144
(005) ret #0

The program can decide to return a value of zero to indicate the packet should be ignored, otherwise it will be kept.

I’ve found that a control flow graph helped me better understand what was going on in BPF programs:

Now that we have a basic understanding of BPF, let’s use it with pcap to sniff HTTP packets.

First, we can use tcpdump to convert the filter ip and tcp and port 80 to BPF bytecode.

$ tcpdump -i en0 -dd "ip and tcp and port 80"
{ 0x28, 0, 0, 0x0000000c },
{ 0x15, 0, 10, 0x00000800 },
{ 0x30, 0, 0, 0x00000017 },
{ 0x15, 0, 8, 0x00000006 },
{ 0x28, 0, 0, 0x00000014 },
{ 0x45, 6, 0, 0x00001fff },
{ 0xb1, 0, 0, 0x0000000e },
{ 0x48, 0, 0, 0x0000000e },
{ 0x15, 2, 0, 0x00000050 },
{ 0x48, 0, 0, 0x00000010 },
{ 0x15, 0, 1, 0x00000050 },
{ 0x6, 0, 0, 0x00040000 },
{ 0x6, 0, 0, 0x00000000 },

We can then use Google’s gopacket package (a pcap wrapper for Golang) to use the above BPF bytecode to sniff HTTP packets.

Once compiled with go build bpf.go, you can run the application as root and start sniffing HTTP packets.

While the above application is quite simple and mostly useless, I learned a bunch about networking and the internals of packet sniffing while building it.

Future readings.

While writing BPF assembly might be fun at first, there are other ways. BPF Compiler Collection (BCC) lets you write BPF programs in a C-like language. There’s a bunch of example apps such as an HTTP packet sniffer.

In newer versions of the Linux kernel, BPF has been enhanced to do a lot more than just packet sniffing like performance analysis. Brendan Gregg of Netflix gave a great talk about this here.

If you are interested in learning more about BPF there is a good doc about it here.