AnyMind Group
Published in

AnyMind Group

Django: detecting common bottlenecks

Computer programmer photo created by pressfoto — www.freepik.com

In this article I won’t be spending much time on performance optimization and best practices, you can check it out here https://medium.com/anymind-group/django-performance-optimization-and-pitfalls-5fc0abed481d.

But rather focus on the tools you can use to detect the most common bottlenecks in your existing app.

Tools:

  1. django-debug-toolbar: configurable set of panels that display various debug information about the current request/response cycle.
  2. django-cprofile-middleware: simple profiling middleware that lets you download profile reports.
  3. snakeviz: browser based profile reports visualizer.

Now let’s take a look at how can we identify some of the issues using the tools above.

SQL related (N+1 queries, over-selecting)

Can be easily detected using django-debug-toolbar SQL panel. Look at the total number of the queries along with the mentions of similar queries.

N+1 queries

If you click on +, you will see a stack-trace and the exact place where the query was executed.

Query stack trace

In this particular case the issue was caused by iterating over books and displaying author’s first name for each book in the template.

The simple fix would be to utilize select_related, it will reduce load on DB and speed up the execution, depending on the latency between the servers the boost will differ a lot; before: 1003 queries, 0.2s, after: 3 queries, 0.04s

Important note: be aware of over-selecting data that is not gonna be used; once your app is big enough and you select_related, prefetch_related many related models in a huge queryset, it might cause some unexpected high memory usage.

Use defer(), only() when necessary and check what is being selected by clicking on Expl (short for explain) button next to the query.

Query explain

For some DB engines explain will also contain Query Plan, where you can quickly estimate size of returned data by multiplying row (amount of rows returned) by width (approximate size of one row in bytes).

Code related

Can be anything, from unnecessary complex nested loops to calling non-optimal methods. To detect the issues you can use django-cprofile-middleware in combination with snakeviz (profile report visualizer).

  1. Download a profile report by adding ?prof&download to the end of URL.
  2. Visualize the report by executing snakeviz path_to_downloaded_report.

You will then see all the methods called, how many times each was called and how long it took.

In this example you can notice that most of the time was spent on apply.

snakeviz: pandas slow apply method

If we google a bit about pandas apply we’ll see that it’s not the most optimal method to call, so we refactor our code a bit to make it faster and check optimized version profile report.

snakeviz: pandas after removing apply method

As you can see it runs much faster now; before: 3.2s, after: 0.1s.

Conclusion

Profiling is no fun, but with the right tools you can ease the process, just give it a try!

Note: the views used in this article can be found here https://github.com/lozhkinandrei/django-common-bottlenecks.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store