Measuring performance of python-based apps using gunicorn and cProfile

Once upon a time one member of my team forgot to get rid of a query somewhere out of Django view context. It fetching 10K large objects from the database. He played with something and left it. Eventually, the piece was deployed to staging and to production afterward. It was not too funny on production. But even after that, it was not obvious how to find an issue.

I used to use debug panels (like Django debug panel) but it’s not flexible enough and there’s no simple way to tweak measurements. Basically, I wanted two things:

  1. Profile my python code
  2. Measure timing for python views (Django or Bottle)

Both are deployed using Gunicorn. And why not to use Gunicorn server hooks in order to measure everything I need?

So, here we go:


Usage:

  1. Save gist above to 「 wsgi_profiler_conf.py
  2. Start gunicorn: 「 gunicorn -c ./wsgi_profiler_conf.py yourapp
  3. Enjoy!

Sample output:


Of course, there’s a lot more useful things like line_profiler and memory_profiler . This approach providing very basic understanding of what is going wrong. The picture above says that most of the time we’re fetching something trough the network. Eventually, we can use tcpdump or wireshark to get an idea of what is going on.