AnyMind Group
Published in

AnyMind Group

[Tech Blog] How to profile code in Python

This article mainly focus on Python and assumes you are already familiar with Python.

Why do we need Python Profilers?

Nowadays, our program is getting more complex in term of both business and coding level which can contain even thousands of code lines. Finding out why the program is slow by looking at its source code is a time consuming task and not easy to do.

Moreover, when the program runs on cloud service which costs more money if the program spends more usage on computing resources.

That’s why code profiler has essential role to find bottlenecks and help to optimize our program performance.

Recently there are several ways to profile code, however in this post, we will be focusing on a basic method using a built-in profilers in Python which is CProfile.

How to use Python cProfile?

cProfile is a built-in python module and is the most commonly used profiler currently, using it we can

  • Get the total run time taken by the entire code.
  • Show the time taken by each individual step which allows to compare and find which parts need optimization.
  • Understand how many times certain functions are being called
  • View stats nicely using other library such as snakeviz

Let’s look at the first simple example

> import cProfile >"sorted([1, 2, 3])") 4 function calls in 0.000 seconds Ordered by: standard name ncalls tottime percall cumtime percall filename:lineno(function) 1 0.000 0.000 0.000 0.000 <string>:1(<module>) 1 0.000 0.000 0.000 0.000 {built-in method builtins.exec} 1 0.000 0.000 0.000 0.000 {built-in method builtins.sorted} 1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
  • ncalls is the number of calls made
  • tottime is a total of the time spent in the given function
  • percall refers to the quotient of tottime divided by ncalls
  • cumtime is the cumulative time spent in this and all subfunctions
  • filename:lineno(function) provides the respective data of each function

There is no bottleneck for the example, let’s look at more interesting example below

import random def sort_letters(number_letters=10**6): random_string = '' for i in range(number_letters): random_string += random.choice('abcdefghijklmnopqrstuvwxyz') return sorted(random_string)

We can also call cProfile on the command line

> python -m cProfile 3000053 function calls in 1.041 seconds Ordered by: standard name ncalls tottime percall cumtime percall filename:lineno(function) 1 0.000 0.000 0.000 0.000<module>) 1 0.000 0.000 0.000 0.000 7 0.000 0.000 0.000 0.000 6 0.000 0.000 0.000 0.000 1 0.001 0.001 0.001 0.001<module>) 1 0.003 0.003 1.041 1.041<module>) 1 0.289 0.289 1.034 1.034 1 0.000 0.000 0.001 0.001 1000000 0.455 0.000 0.562 0.000 1 0.002 0.002 0.004 0.004<module>) ...

By looking at the output, it looks like the choice function from random module takes lot of time, we can start optimizing it from this code

How to visualize cProfile reports?

The common tool for visualizing data obtained by cProfile module is SnakeViz, let explore how to use it

Firstly, we can export the output of cProfile in to the dump file

python -m cProfile -o stats.dump

Next step

snakeviz stats.dump

Start optimizing previous code

From previous example, we know that random.choice function is slowing us down, it's time to replace it with other similar function

from numpy import random def sort_letters(number_letters=10**6): letters = tuple('abcdefghijklmnopqrstuvwxyz') random_letters = random.choice(letters, number_letters) random_letters.sort() return random_letters.tostring()

Now our program runs much more faster than before


Apart from cProfile, there are some other python module to support profiling code, but cProfile includes most of its features and is recommended to use. Stay tuned for similar topic, for example server-side profiler

Originally published at

AnyMind Group has created one of the first end-to-end brand enablement platforms, developing and providing inter-connected software and offerings to enable businesses and individuals — from brand building, manufacturing and e-commerce, to marketing, logistics and global expansion

Recommended from Medium

DS With JS — Hash Tables

How to start your Ghost Blog for almost FREE in cloud

Case Study Presentation Grading Rubric

🔎 New Airdrop : Polymer Finance 💸 Token Value : $10 PMF ⏳ Duration Time : 4 Minute 👥 Referral …

Hướng dẫn cách đọc 6 tỷ lệ cược bóng đá Win365 kèo Châu Á phổ biến nhất

Jackson API beyond ObjectMapper

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store


More from Medium

MongoDB REplication

Migration Handling While using schema based multi-tenancy in Flask application

KWOC | Kharagpur Winter Of Code Project Report

Working with database system: Postgresql