Groking Python decorators: a beginner’s guide

I was recently asked by a friend to explain how Python decorators work. By any estimation, decorators are one of those slippery object oriented programming concepts that, understandably, takes the unfamiliar more than just a minute to fully understand.

(Before you read further, this post assumes you have a basic understanding of how functions, variable scope, and *args and **kwargs work).

Consider the following code:

def p_decorate(func):
   def func_wrapper(*args, **kwargs):
       return "<p>{0}</p>".format(func(*args, **kwargs))
   return func_wrapper
@p_decorate
def get_text(name):
   return "lorem ipsum, {0} dolor sit amet".format(name)

print get_text("chip")
# get_text("chip") outputs:
# <p>Outputs lorem ipsum, chip dolor sit amet</p>

Ok, so what’s happening here?

First off, we need to get some clarity on just exactly what a decorator is:

A decorator is any callable Python object that is used to modify a function, method or class definition. A decorator is passed the original object being defined and returns a modified object, which is then bound to the name in the definition. Python decorators were inspired in part by Java annotations, and have a similar syntax; the decorator syntax is pure syntactic sugar, using @ as the keyword.

So we can boil that down even further by saying simply that: a decorator is just a function that takes another function and returns a new (decorated or changed) function.

This is different from what a beginner normally is used to seeing, wherein a function is a thing that returns a value, a final result. A decorator doesn’t return a final return value “end product” but instead returns a new, augmented function, that is then used to complete a task. If we think of functions like little machines, then a decorator is like a factory that takes in little machines, makes alterations, and then returns the altered machine to the user for further use.

So in our code above, the first weird thing you see is the @p_decorate

The use of the @ symbol is just syntactic sugar and means the same thing as if we were to write this instead:

my_get_text = p_decorate(get_text)

So, just to be clear, without the @ symbol, the code above could be written just as easily (but not Pythonically) as this:

def get_text(name):
   return "lorem ipsum, {0} dolor sit amet".format(name)

def p_decorate(func):
   def func_wrapper(*args, **kwargs):
       return "<p>{0}</p>".format(func(*args, **kwargs))
   return func_wrapper

my_get_text = p_decorate(get_text)

print my_get_text("chip")

Ok, so back to the original code at the top of the post with the @ symbol.

Let’s break this down:

The function we’re decorating, or augmenting the functionality of, is get_text(name). This is a pretty simple function that we pass a string to as the argument and it returns a new string with the name inserted into the Latin text: “lorem ipsum, chip dolor sit amet”.

Our decorator function is p_decorate(func) and you should note its construction:

def p_decorate(func):
   def func_wrapper(*args, **kwargs):
       return "<p>{0}</p>".format(func(*args, **kwargs))
   return func_wrapper

The p_decorate(func) has a nested or inner function inside of it called func_wrapper() which takes *args and **kwargs (more on that in a minute).

Think of the decorator p_decorate() as a “function factory” which exists to house/create/and most importantly return another function. What is that other function? It’s the altered version of the function that was decorated. How the decorated function, in this case get_text(name), get’s ‘decorated’ or altered is via the inner or nested function inside of p_decorate() called func_wrapper(), which is (as its name suggests) a function wrapper, because it ‘wraps’ around another function. That function, is of course, the decorated function get_text().

Inside of the function wrapper func_wrapper(*args, **kwargs) we can see that we’re returning the value returned by func(*args, **kwargs) (which is a string) inserted into the html paragraph tags “<p>{0}</p>”.

Note that func(*args, **kwargs) is actually get_text(“chip”), this is crucial, keep in mind everything in Python is an object and all we’ve done here is pass one function into another function.

Confused? Remember:

@p_decorate
def get_text(name):
....
#is the same as saying:
my_get_text = p_decorate(get_text) 

So you can see, we’re passing to p_decorate(func) the get_text(name) function. So inside of p_decorate(func) just keep in mind that func == get_text(name). Just a bit of syntactical groking housekeeping there, but it’s worth pointing out.

But what else is happening inside of p_decorate(func)? Well, it’s the most important concept to understand about decorators: what p_decorate(func) is actually returning to the global namespace of the script is NOT a value but ANOTHER FUNCTION! Again, this is what decorators do:

def p_decorate(func):
    def func_wrapper(*args, **kwargs):
        return "<p>{0}</p>".format(func(*args, **kwargs))
    return func_wrapper

Look at the code above: p_decorate(func) is returning the func_wrapper function object back to the global namespace!!! What is func_wrapper after all? Well, it’s essentially our get_text(name) function with some added functionality (the <p> tag formatting)!

So when we do print get_text(“chip”) what we’re actually calling — because we decorated def get_text(“chip”) with @p_decorate — is func_wrapper(“chip”).

Let that sink in for a minute.

Again:

@p_decorate
def get_text(name):
....
#is the same as saying:
my_get_text = p_decorate(get_text)

So our decorated get_text(name) function has actually become func_wrapper(name) simply by decorating get_text(name).

A few things to note:

There’s an important concept at work within p_decorate(func) called closure. Without getting too off track here, it’s important to make mention of this phenomenon in Python.

Basically A CLOSURE is a function object that remembers values in enclosing scopes regardless of whether those scopes are still present in memory. Python supports a feature called function closures which means that inner functions (like our func_wrapper(*args, **kwargs)) defined in non-global scope remember what their enclosing namespaces (the namespace or scope “owned” by p_decorate(func)) looked like at definition time.

This can be seen by looking at the func_closure attribute of our inner function which contains the variables in the enclosing scopes:

...
my_get_text = p_decorate(get_text)
print my_get_text.func_closure
# this print statement would return something like this:
# (<cell at 0x108de92b8: function object at 0x1089296e0>,)

But don’t worry about all that for now.

The other important thing here is: why are we using *args and **kwargs in our func_wrapper() function?

In order for func_wrapper() to be as useful and flexible as possible we’d like it to accept any kind of arguments that could possibly be thrown its way, right? That’s why we’re using *args and **kwargs — it’s just good practice to make our decorator function as flexible as possible, which means that we’d prefer to not hard-code arguments into it, so that way we can use it all over the place in our code and get the same functionality without having to go back and tweak it every time.

Things can get a little more complicated with the use of functools module or if you have a decorator with an argument. I’ll try to follow up on that with another post.