Advanced Python Learning: Optimisation of Code Using Profilers

Image By Author

Python is a powerful and widely-used programming language that is known for its simplicity and ease of use. However, like any other language, Python programs can sometimes run slowly or consume more resources than intended. This can be a problem if you are working on a project that needs to process large amounts of data or handle high volumes of traffic. Fortunately, Python provides a variety of tools that can help you identify performance issues and optimize your code.

In this blog post, we’ll discuss Python profilers and how they can help you improve the performance of your Python programs.

What is a Profiler?

A profiler is a tool that measures the performance of a program by analyzing its execution behavior. Profilers can help you identify bottlenecks and other performance issues that may be slowing down your program. Profilers can be used to analyze the performance of a program at different levels, including function level, line level, and statement level.

Python provides a variety of profilers that can help you analyze the performance of your programs. Some of the most popular profiles are built-in to the Python standard library, while others are available as third-party libraries such as cProfile, profile, PyCharm Profiler, etc,

“Optimizing code without profiling is like driving with your eyes closed.” — Paul Bissex

Let deep dive profilers from two friends

Manu and Teja were discussing a problem with a Python code that Manu had been working on. The code was taking a long time to execute and Manu suspected that there might be some performance issues. That’s when Manu told Teja about profilers.

Manu: Have you heard of profilers, Teja?

Teja: No, what are they?

Manu: Profilers are tools that help you identify performance issues in your code. They tell you which functions are taking the most time to execute, so you can optimize them for better performance.

Teja: That sounds useful. Can you give me an example of how it works?

Manu: Sure, let me show you. Here’s a simple Python program that calculates the sum of squares of numbers from 1 to n using a loop.

def sum_of_squares(n):
    result = 0
    for i in range(1, n+1):
        result += i*i
    return result

if __name__ == '__main__':
    print(sum_of_squares(100000))

Teja: That looks pretty straightforward. But what if the value of n is really large? Wouldn’t it take a lot of time to execute?

Manu: Yes, that’s exactly what we’re going to test. Let’s run this code with a significant value of n and see how long it takes.

import time
start_time = time.time()

print(sum_of_squares(1000000))
end_time = time.time()
print(f"Time taken: {end_time - start_time} seconds")

Teja: Wow, that took a long time to execute! Is there any way to speed it up?

Manu: Yes, that’s where profilers come in. Let’s run this code through cProfile and see what it tells us.

def sum_of_squares1(n):
    total = 0
    for i in range(1, n + 1):
        total += i * i
    return total

if __name__ == "__main__":
    start_time = time.time()
    result = sum_of_squares1(100000)
    end_time = time.time()
    print("Result: ", result)
    print("Execution time: ", end_time - start_time, "seconds")
cProfile.run('sum_of_squares1(100000)')

Result:  333338333350000
Execution time:  0.008126258850097656 seconds
         100006 function calls in 0.016 seconds

   Ordered by: standard name
   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.016    0.016 <string>:1(<module>)
        1    0.000    0.000    0.016    0.016 testing_profiler.py:5(sum_of_squares)
   100001    0.009    0.000    0.009    0.000 testing_profiler.py:6(<genexpr>)
        1    0.000    0.000    0.016    0.016 {built-in method builtins.exec}
        1    0.007    0.007    0.016    0.016 {built-in method builtins.sum}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

Teja: That’s really helpful! It looks like the majority of the time is being spent on the __mul__ function call.

Manu: Exactly. Now that we know which function is causing the delay, we can optimize it for better performance. Let’s try using a list comprehension instead of a loop and see how it affects the performance.

def sum_of_squares(n):
    return sum([i*i for i in range(1, n+1)])

if __name__ == '__main__':
    print(sum_of_squares(1000000))

Teja: That’s much faster! How much time did it save?

Manu: Let’s find out.

import time
import cProfile

def sum_of_squares(n):
    return sum(i * i for i in range(1, n + 1))

if __name__ == "__main__":
    start_time = time.time()
    result = sum_of_squares(100000)
    end_time = time.time()
    print("Result: ", result)
    print("Execution time: ", end_time - start_time, "seconds")
cProfile.run('sum_of_squares(1000000)')

Result:  333338333350000
Execution time:  0.006894111633300781 seconds
         4 function calls in 0.008 seconds

   Ordered by: standard name
   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.008    0.008 <string>:1(<module>)
        1    0.008    0.008    0.008    0.008 testing_profiler.py:19(sum_of_squares1)
        1    0.000    0.000    0.008    0.008 {built-in method builtins.exec}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

Teja: That’s a huge improvement! We went from over 6 seconds to less than a second.

Manu: Exactly. That’s the power of profilers. By using them, we were able to identify the bottleneck in our code and optimize it for better performance.

Advantages of using a profiler:

1. Identify performance bottlenecks: Profilers can help you identify which parts of your code take the most time to execute. This can help you focus your optimization efforts on the details of your code that will have the greatest impact on performance
2. Optimize your code: Once you have identified the performance bottlenecks in your code, you can work on optimizing those parts of your code to improve the overall performance of your program.
3. Understand the behavior of your code: Profilers can help you understand the behavior of your code by showing you which functions are being called, how long they are taking to execute, and how often they are being called
4. Save time: Profilers can save you time by helping you quickly identify and fix performance issues in your code. This can be especially important if you are working on a large codebase
5. Improve user experience: By optimizing the performance of your code, you can improve the user experience of your program. This can lead to happier users and better reviews

“Profiling is the process of analysing your code to identify performance bottlenecks and optimising it to improve its performance.” — ChatGPT

Conclusion:

Profiling is a crucial step in optimizing your Python code. It helps you identify performance issues and bottlenecks in your code, so you can optimize them for better performance. Python has several profilers available, including cProfile and PyCharm.