Django: detecting common bottlenecks
In this article I won’t be spending much time on performance optimization and best practices, you can check it out here https://medium.com/anymind-group/django-performance-optimization-and-pitfalls-5fc0abed481d.
But rather focus on the tools you can use to detect the most common bottlenecks in your existing app.
Now let’s take a look at how can we identify some of the issues using the tools above.
SQL related (N+1 queries, over-selecting)
Can be easily detected using
django-debug-toolbar SQL panel. Look at the total number of the queries along with the mentions of similar queries.
If you click on
+, you will see a stack-trace and the exact place where the query was executed.
In this particular case the issue was caused by iterating over books and displaying author’s first name for each book in the template.
The simple fix would be to utilize
select_related, it will reduce load on DB and speed up the execution, depending on the latency between the servers the boost will differ a lot; before: 1003 queries, 0.2s, after: 3 queries, 0.04s
Important note: be aware of over-selecting data that is not gonna be used; once your app is big enough and you
prefetch_related many related models in a huge queryset, it might cause some unexpected high memory usage.
only() when necessary and check what is being selected by clicking on
Expl (short for explain) button next to the query.
For some DB engines explain will also contain
Query Plan, where you can quickly estimate size of returned data by multiplying
row (amount of rows returned) by
width (approximate size of one row in bytes).
Can be anything, from unnecessary complex nested loops to calling non-optimal methods. To detect the issues you can use
django-cprofile-middleware in combination with
snakeviz (profile report visualizer).
- Download a profile report by adding
?prof&downloadto the end of URL.
- Visualize the report by executing
You will then see all the methods called, how many times each was called and how long it took.
In this example you can notice that most of the time was spent on
If we google a bit about pandas
apply we’ll see that it’s not the most optimal method to call, so we refactor our code a bit to make it faster and check optimized version profile report.
As you can see it runs much faster now; before: 3.2s, after: 0.1s.
Profiling is no fun, but with the right tools you can ease the process, just give it a try!
Note: the views used in this article can be found here https://github.com/lozhkinandrei/django-common-bottlenecks.