How to create a cprofile decorator class and save stats in Docker Container— Python

In this post I wanted to explain how I built my cprofile class for saving stats and check the runtime of some functions I was using in my project.

As we all know, cprofile let is analyse the time each process takes or other metrics in your project. In order to reduce most of the times the runtime of a determined functions. Find the flaws and improve them or fix them.

The project I was developing is bigger but I took the part of the profiling to show you guys the specific steps to reproduce it.

First of all we create a docker Dockerfile as simple as this:

FROM archlinux:20200407
LABEL "Author"="marco.maigua1346@gmail.com"
ENV container docker
COPY docker/08-onboard_api/entrypoint.sh /
RUN chmod +x /entrypoint.sh
CMD ["/entrypoint.sh"]

The entrypoint.sh file will create the directory where our logs will be

#!/bin/bash
if [[ ! -f /var/log/profiling ]]; then
    # Session Stats folder
    mkdir -p /var/log/profiling
    chown row44:log /var/log/profiling
    chmod 744 /var/log/profiling
fi

Thereafter the following class Diagnostic defines a decorator that can be used in some other class. Therefore any call to the method that has the decorator will pass over these instructions to save the run time stats.

You can see the parameters of cprofile, and the settings of the stats format in here:

import cProfile
import pstats

from onboard_api.libservice.utils.format import get_output_file

is_profiling = True
sort_name = 'cumtime'
path = "/var/log/onboard_api_profiling"

class Diagnostic:
    def __init__(self):
        pass

    @staticmethod
    def diagnostic_with_cprofile_decorator():
        """
        description: cprofile_diagnostic decorator
        required: cProfile and pstats
        sort_name: string > options:
            calls (call count)
            cumulative (cumulative time)
            cumtime (cumulative time)
            file (file name)
            filename (file name)
            module (file name)
            ncalls (call count)
            pcalls (primitive call count)
            line (line number)
            name (function name)
            nfl (name/file/line)
            stdname (standard name)
            time (internal time)
            tottime (internal time)

        def run(func):
            def wrapper(*args, **kwargs):
                if is_profiling:
                    return Diagnostic.profile_output(func, args, kwargs)
                else:
                    return func(*args, **kwargs)

            wrapper.__doc__ = func.__doc__
            return wrapper

        return run

In this same class I added the method profile_output that will take charge of the writing of the stats over the files created in the entrypoint.sh in the Dockerfile

Something additionally I added is is_profiling flag so we can decide if we want to get the stats or not. Sometimes we just want to run it once to diagnose code that can be improved.

    @staticmethod
    def profile_output(func, args, kwargs):
        # get the class name
        output_file = get_output_file(sort_name, path)
        prof = cProfile.Profile()
        prof.enable()
        ret_val = prof.runcall(func, *args, **kwargs) # << attention
        prof.disable()
        with open(output_file, "a+") as f:
            ps = pstats.Stats(prof, stream=f)
            ps.sort_stats(sort_name)
            ps.print_stats()
        return ret_val

I took the freedom of creating a method that gets the generic path of depending on what we need.

def get_out_file(sort_name, path):
    output_file = path + ".log".format(sort_name)
return output_file

An example of the use of this class could be:

from diagnostic_cprofile import diagnostic_with_cprofile_decorator

    class MyClass(object):

        @diagnostic_with_cprofile_decorator('cumtime')
        def my_method(self):
        ...

Finally, if we need to call a utility class, that is not precisely part of a bigger class, can just call to the method like this:

@staticmethod
    def diagnostic_with_cprofile_utils(func, *args, **kwargs):
        """
        description: cprofile_diagnostic utils
        required: cProfile and pstats
        sort_name: string > options:
            calls (call count)
            cumulative (cumulative time)
            cumtime (cumulative time)
            file (file name)
            filename (file name)
            module (file name)
            ncalls (call count)
            pcalls (primitive call count)
            line (line number)
            name (function name)
            nfl (name/file/line)
            stdname (standard name)
            time (internal time)
            tottime (internal time)

        """
        if is_profiling:
            return Diagnostic.profile_output(func, args, kwargs)
        else: 
            return func(*args, **kwargs)

As an example of the use of this method for utility classes, we can use it like this:

from diagnostic_cprofile import diagnostic_with_cprofile_utils

    items = diagnostic_with_cprofile_utils(get_content_items, content_filter,custom_arg=True, custom_arg='bool', .....)

At the end we will have file formats such as this ones in the path we specified for this example /var/log/sessions_stats

I hope this makes sense, is very straight forward. You can change the format of cprofile as you require as well. Let me know if you have any questions. I will be happy to help :)

my email: marco.maigua1346@gmail.com