Solving a Tricky Problem

What to do when Google fails you

Every once in a while when you go to Google an issue you’re having, you don’t find any good results. After closing the tab, re-opening it, and searching for the same thing 2 or 3 more times before realizing what you’re doing, you might be tempted to give up.

Should you quit? Hell no! Unless you don’t feel like it. Then you should quit. But this time I didn’t. So here’s the story of a time when I didn’t quit. It even has a happy ending.

The Problem

A couple of days ago, I ran into a problem like this. I was trying to debug an SSL connection issue that only showed up sporadically. I used tcpdump combined with a looping bash script to try to find the issue and get a network capture of it.

for i in $(seq 60); do
    gtimeout 45 ./run.sh > /tmp/capture-$i.log &
    gtimeout 45 tcpdump -i any -w /tmp/capture-$i.pcap ‘src 1.1.1.1 || dst 1.1.1.1’ &
     wait
done

Basically, I was just running the script I was having trouble with along with tcpdump for 45 seconds, waiting for both to end, and logging to a file that would show the error if it happened, as well as a pcap.

After I finally managed to reproduce the problem and get a capture, I was disappointed to see this in wireshark:

The capture file appears to be damaged or corrupt. (pcapng: interface index 1 is not less than interface count 1)

Awesome. Well maybe I could take a look at the data in tcpdump at least:

reading from PCAP-NG file capture-41.pcap
tcpdump: pcap_loop: a packet arrived on interface 1, but there’s no Interface Description Block for that interface

Unacceptable. This was a rare capture; I needed to figure out a way to take a look at it. Hell, even if I managed to capture it again, there was no guarantee it wouldn’t just be corrupt again.

The Tricky Road to a Solution

The first thing I noticed was tcpdump thought this file was a “pcap-ng” file, rather than a pcap file. I looked up the difference between the two and found that pcap-ng files contain extra meta information about things like the interface! Perhaps if I could convert to the less-information pcap format, it would simply strip this extra information out and then be readable.

According to The Wireshark Wiki, there are a couple of tools that can be used to do this conversion. So I tried the first one:

dan@Dans-MacBook-Pro:/tmp$ editcap -F libpcap -T ether capture-41.pcap out
(process:4447): GLib-CRITICAL **: g_hash_table_lookup: assertion `hash_table != NULL’ failed
editcap: An error occurred while reading “capture-41.pcap”: The file appears to be damaged or corrupt.
(pcapng: interface index 1 is not less than interface count 1)

What a bunch of gibberish! What a bunch of crap! What a bunch of — okay, I needed to stay calm. Maybe the second tool would help:

dan@Dans-MacBook-Pro:/tmp$ tcpdump -r capture.pcap -w out
reading from PCAP-NG file sc-41.pcap
tcpdump: pcap_loop: a packet arrived on interface 1, but there’s no Interface Description Block for that interface

Well it was still tcpdump, so I wasn’t sure what I expected.

My next step was to see if there were any libraries for parsing pcapng files I could leverage. Since I had a vague notion of what the problem might be (I had a feeling it was related to the filtering I added to tcpdump), maybe I could use an abstraction over the file format to tinker with it and fix the issue.

I ended up finding python-pcapng. I started by just loading up the file and looking around:

Looking at the first block (presumably the header), I found this:

Okay, one interface with an index of 0. That made sense based on the error. Next, I found the first packet:

Ah! It was referencing the interface as 1, rather than 0. Trying to access the interface information for the packet confirmed my suspicion:

If I could just change this on all of the packets and re-write the file, perhaps that would fix the problem. Unfortunately, I went back to the python-pcapng page and saw something unfortunate:

Support for writing pcap-ng files is “planned”; [..] I didn’t add that part (yet) as I currently don’t need it, and I’m wondering whether anybody might.

Damn. Still, it wasn’t the time to give up. Thankfully, earlier in the readme for the project the pcapng file format spec is referenced. Maybe I would be able to find what I needed there.

Indeed, after looking around a bit, I stumbled upon a nice diagram for the so-called “Enhanced Packets”:

Enhanced Packet Block format.

I must confess, I didn’t anticipate the day when a block ascii diagram would get me really excited, but it doesn’t get a lot more self-explanatory than this. My next step was to open up the capture in a hex editor and see if I could by hand find the packet that python-pcapng had showed me.

After searching for the payload shown in the above example, I was able to find the start of the packet (referenced in the block diagram as being 4 bytes, starting with 0x06):

Looking at the second block from the start of the beginning of the packet (4 bytes later), I saw something that both lined up with “Interface ID” in the block diagram and contained the data:

01 00 00 00

Bingo!

The next step was to replace that 01 with a 00 throughout the file. The only problem was that block in the middle. According to the block diagram, it specifies “Block Total Length,” which sounds pretty important. Unfortunately, my hex editor didn’t have any support for capturing groups or wildcards for search and replace, so it was back over to Python.

I popped open a shell and whipped up a little big of regex to patch the file:

Now for the moment of truth. Would it work?

Success! That felt good.

What Next?

Maybe now I’ll try to reproduce the problem again with tcpdump and file an issue. Maybe I’ll try to fix it myself. Maybe others running into this issue will now be able to find it on Google. Who knows? Like they always say:

Once you’ve hand-patched a corrupted pcapng file using a mixture of regexes, python, and a hex editor, the world is your oyster.