How to use ngrep to debug HTTP headers

You deploy Django with gunicorn. Your web server communicates with it with proxy_pass (nginx) or with ProxyPass (apache), together with several other directives that pass headers and so on. Your web site doesn’t work properly. Is the web server misconfigured, or is Django misbehaving? Here’s how to find out.

Install ngrep:

apt-get install ngrep

Now try this:

ngrep -W byline '' 'tcp port 80'

ngrep intercepts communications at port 80; that is, between the web server and the web browser, so whenever a request is made, you will be viewing the request and the response. Press Ctrl+C to stop ngrep.

The -W byline part has to do with how ngrep formats its output (try it without it to see the difference). The next argument, the empty string, tells ngrep to output all communication; if it were non-empty it would only show lines that match it (for example, if you try nginx -W byline ': ' 'tcp port 80', it won’t catch much more than HTTP header lines).

If your web server has traffic, there is going to be much output. You want to intercept only requests from your own browser. In that case, use this:

ngrep -W byline '' 'tcp port 80 and host [your ip address]'

(An easy way to find your ip address is to type who -m --ips at the remote command prompt.)

Now, assuming that gunicorn is listening on port 8000, try this:

ngrep -W byline -d lo '' 'tcp port 8000'

Now you are intercepting the communication between the web server and gunicorn. The -d lo tells ngrep to intercept the local network interface (i.e. a virtual network device that connects the system to itself) instead of the external ethernet interface (i.e. the actual network device), which is what you want if gunicorn has been told --bind 127.0.0.1:8000 (the 127.0.0.1 is the local interface).

Unfortunately, in this case I don’t know any way to filter the output so that only the requests from your browser are allowed. On servers with little traffic I can sometimes start ngrep, make the request in my browser, and stop ngrep; if I repeat that a few times, I can often manage to catch only my request. If the server has much traffic, you need either to use a staging server, or, if you can afford the downtime and the risk of playing with a firewall on a production system, to temporarily filter out all requests but yours with a firewall.

This is a republication of an old article first published in my Django Deployment site.