Daniil Ekzarian
Feb 4, 2018 · 3 min read

Good evening everyone!
Today I’ve faced a problem with my Python web app, which required from me to trace the memory usage to find the possible memory leak. I’ve learned some methods and libraries for monitoring memory traffic and I want to share those.

computer-memory-chips
computer-memory-chips

One of the basic libraries for getting memory usage information is a resource. This is default Python distribution library, so you can just import it.

Let’s see how it can be used:

mem_a = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss/1000
print("Memory overused %.2fKB" % (mem_a - mem_b))
  • resource.RUSAGE_SELF — shows resource usage for all process threads;
  • ru_maxrss — maximum resident set size

Unfortunately, this method will not give a lot of information, because it can only show the amount of memory that is owned by process or thread at a particular time without any insights.

Note that if your program is generating a lot of memory traffic it will be owned by the process until the end of process execution!

For more detailed memory view, you can use objgraph. This library is not the default one, so `pip install objgraph` first.

To show graphs objgraph requires graphviz to be installed.

Let’s first see how objgraph can generate visual representations of memory links.

import objgrapha = [1,2,3, [1,2, [1,3]]]
objgraph.show_refs([a], filename='graph.png')

This code will generate 'graph.png' image:

graph
graph

Using these simple methods we can create complex methods of memory analysis.

My idea is to create a decorator which tries to find leaks of specific object type and if found, shows back references of those.

We'll assume that we have a function, that shouldn't have any side effects of the particular object type.

This will be our object type:

class MyLeakyObject:
a = []

For memory testing, I've written this memory_leak decorator:

def memory_leak(t):
def inner(f):
def wrapper(*args):
objgraph.growth()
r = f(*args)
after = objgraph.growth()
leaks = list(filter(lambda x: x[0] == t.__name__, after))
if leaks:
print('Memory leaks!')
objgraph.show_chain(
objgraph.find_backref_chain(
random.choice(objgraph.by_type(t.__name__)),
objgraph.is_proper_module),
filename = 'chain.png')
else:
print('No leaks were found!')
return r
return wrapper
return inner

First, we'll test function with no memory leaks:

@memory_leak(MyLeakyObject)
def function_without_leak():
a = MyLeakyObject()
# Do smth

This function prints ​No leaks were found!.

In the second case, we store object to a global cache.

cache = []
@memory_leak(MyLeakyObject)
def function_with_leak():
a = MyLeakyObject()
# Do smth
cache.append(a)

The output is `Memory leaks!
Graph written to /var/folders/f7/5tc8dsc1593306gbpjm07zmw0000gn/T/objgraph-x0qe7rh9.dot (4 nodes)
Image generated as chain.png​`.

chain
chain

You can also use guppy for analysing the heap contents, but unfortunately, it works only with Python 2.X.

For tracing WSGI applications there is a good memory leak debugger Dozen, which is capable of showing traces of requests with nice visualization using GraphViz.

Links

1 resource module documentation - https://docs.python.org/3/library/resource.html

2 object graph - https://mg.pov.lt/objgraph/

3 Dozen - https://pypi.python.org/pypi/Dozer

4 Dozen usage guide - https://mg.pov.lt/blog/profiling-with-dozer.html

Reflash Programming Adventures

My adventures with programming

Daniil Ekzarian

Written by

Software engineer 👨‍💻 Functional programming disciple

Reflash Programming Adventures

My adventures with programming

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade