ADVANCED PYTHON PROGRAMMING

To Functions, and Beyond!

This time, we consider functions as objects — and develop the decorator design pattern on top of that, in all its glory.

11 min readApr 8, 2020

Now that we know how to write functions and call them, let’s pause for a moment and talk about what they actually are; because in Python, everything is an object—and that includes functions. They can be passed on as arguments, returned as return values, have attributes, and take part in interesting software compositions, such as the decorator design pattern.

What Maketh an Object?

We’ll talk about objects in great detail in later posts—for now, we’ll settle for the formal definition of, “um, well, a thing you can.. uh, do stuff with?”. In light of this, we’ll try to understand whether Python functions qualify as objects or not.

Pitchin’

One thing you can do with objects is pass them on as an argument to some function. Passing a function to another function might sound cannibalistic, but it’s definitely doable (and even useful—but more on that later):

>>> def run_twice(f):
...     f()
...     f()>>> def hello():
...     print('Hello, world!')>>> run_twice(hello)
Hello, world!
Hello, world!

Catchin’

Another thing you can do with objects is get them back from a function. Receiving a function from another function might sound… surprisingly biologically viable, actually—but in any case, it’s also doable:

>>> def create_power(n):
...     def power(x):
...         return x**n
...     return power>>> square = create_power(2)
>>> square(2)
4>>> cube = create_power(3)
>>> cube(2)
8

Functions as Objects

Another thing objects have in common is that they’re all “object-y”. They have some type, some representation, and possibly attributes—either built-in ones or custom ones. Well, functions do, too:

>>> def add(x, y):
...     "Returns the sum of x and y."
...     return x + y>>> add
<function add at 0x...>>>> type(add)
<class 'function'>>>> add.__name__
'add'>>> add.__doc__
'Returns the sum of x and y.'>>> add.x = 1
>>> add.x
1
>>> del add.x

So as it turns out, you can get a function’s name, docstring and other internal details (like its code object) through special attributes, beginning and ending with a double underscore (pronounced “dunder-X”, as in “dunder-name”). You can also get, set and delete your own attributes, which allows for the occasionally useful function state:

>>> def hello():
...     hello.runs += 1
...     print('Hello, world!')
>>> hello.runs = 0>>> hello()
Hello, world!
>>> hello()
Hello, world!
>>> hello.runs
2

Note that there’s nothing special about this state: having talked about scopes, we know that when the hello function runs, it simply tries to resolve the non-local name hello, reaches the scope of its own definition, gets a pointer to itself, and tickles its run attribute; nothing more, nothing less.

Decorum

So then, let’s talk about decorators: a decorator is a function that receives a function and returns a function, usually decorating it with some additional functionality in the process. If that seems like a lot, just see for yourself:

>>> def double(f):
...     def wrapper(x):
...         return 2 * f(x)
...     return wrapper>>> def inc(x):
...     return x + 1>>> inc = double(inc)
>>> inc(1)
4 # (1 + 1) * 2

In this story, double was the decorator: it received the function inc as an argument (for the parameter f), defined a new function, wrapper, and returned it in its stead. This new function is defined with inc in its scope (again—as f), so whatever the original function, it can invoke it and double its result. The weird inc = double(inc) notation is a common idiom of rebinding the function’s name to this “enhanced” version of itself; in fact, it’s so common it even has its own syntactic sugar:

>>> @double
... def inc(x):
...     return x + 1>>> inc(1)
4

The @double notation takes whatever’s defined underneath it, passes it through double, and rebinds the result to the same name—in this case, inc.

The Omnisignature Strikes Back

Of course, this isn’t all that useful—the double function was tailor-made to create the wrapper function with exactly one argument, x, because inc is a function that receives exactly one argument, x. Had it been a function with any other signature, this wouldn’t have worked. If only we had a signature that’d always work, and some way to forward it perfectly…

def double(f):
    def wrapper(*args, **kwargs):
        return 2 * f(*args, **kwargs)
    return wrapper

This decorator is much more flexible: it replaces f with some wrapper of it, which receives whatever—and passes it on, exactly as it is, to the original f (and then doubles the result, of course).

This is a bit of a silly example, but it works well for illustration purposes; now that you’re familiar with the syntax, let’s see some real examples.

Tracing

I’m one of those old-fashioned people that don’t use a debugger, but prefer print statements. That means I’d often like to be able to trace my program’s execution: you know, which function gets invoked when, with what arguments, and to to what ends. Instead of adding logging before and after every piece of indented code, I can abstract this so-called “cross-cutting concern” into an external utility—and then apply it to my functions using the decorator design pattern, like so:

def trace(f):
    def wrapper(*args, **kwargs):
        print(f'enter {f.__name__}')
        try:
            return f(*args, **kwargs)
        finally:
            print(f'leave {f.__name__}')
    return wrapper

This can then decorate any function as easily as adding @trace to its crown:

>>> @trace
... def div(x, y):
...     return x / y>>> div(4, 2)
enter div
leave div
2.0
>>> div(1, 0)
enter div
leave div
Traceback (most recent call last):
  ...
ZeroDivisionError: division by zero

This is nice, but we can make it even cooler:

def trace(f):
    def wrapper(*args, **kwargs):
        params = []
        if args:
            params.extend(repr(a) for a in args)
        if kwargs:
            params.extend(f'{k}={v!r}' for k, v in kwargs.items())
        call = f'{f.__name__}({", ".join(params)})'
        print(f'enter {call}')
        try:
            result = f(*args, **kwargs)
            print(f'leave {call}: {result!r}')
            return result
        except Exception as error:
            print(f'leave {call} on error: {error}')
            raise
    return wrapper

This way, we also get the arguments, return values and raised exceptions!

>>> @trace
... def div(x, y):
...     return x / y>>> div(4, 2)
enter div(4, 2)
leave div(4, 2): 2.0
2.0
>>> div(1, 0)
enter div(1, 0)
leave div(1, 0) on error: division by zero
Traceback (most recent call last):
  ...
ZeroDivisionError: division by zero

This example is great for debugging; but if it’s not a compelling enough reason, here’s a use case that can speed your code up considerably—and everyone wants performance, right?

Caching

Let’s say you’ve implemented the Fibonacci function with recursion:

def fib(n):
    return n if n < 2 else fib(n - 1) + fib(n - 2)

Go ahead and try to compute the 50th Fibonacci number — on my Mac, it took a whopping hour and a half to do so. But that doesn’t have to be the case! After all, fib(50) only ever computes 50 numbers—it’s just recomputing the same numbers again and again. With fib(50) = fib(49) + fib(48), and fib(49) = fib(48) + fib(47), it means fib(49) gets computed twice; imagine (or calculate!) how many times fib(1) gets computed.

Instead of letting this tomfoolery continue, we could cache the values that were already computed, and fetch them from our cache instead of recomputing them whenever they’re needed. This technique is called memoization, and fits a wide variety of problems; so instead of solving it just for Fibonacci, let’s solve it once and for all with a decorator that caches return values based on the arguments:

def memoize(f):
    cache = {}
    def wrapper(*args, **kwargs):
        token = args + tuple(kwargs.items())
        if token not in cache:
            cache[token] = f(*args, **kwargs)
        return cache[token]
    return wrapper

Apply it to your function, and bam:

>>> @memoize
... def fib(n):
...     return n if n < 2 else fib(n - 1) + fib(n - 2)>>> fib(50)
12586269025

Less than a second!

Cheap Wrapping Paper

Of course, nothing’s perfect—not even decorators. One of their unfortunate side effects is that the original function gets replaced by another—with a different name, and a different docstring. You might not care much, but auto-documentation tools sure do:

>>> @double
... def add(x, y):
...     """Returns the sum of x and y."""
...     return x + y>>> add.__name__
'wrapper'
>>> add.__doc__
''

Luckily, Python comes with the built-in functools.wraps decorator, which you simply apply to your wrapper to make all your problems go away. This might melt you brain, but bear with me—we’ll implement it ourselves in a moment.

>>> import functools
>>> def double(f):
...     @functools.wraps(f)
...     def wrapper(*args, **kwargs):
...         return 2 * f(*args, **kwargs)
...     return wrapper

That’s… a lot. I mean, there’s a function call inside a decorator, which is used to decorate the wrapper returned from another decorator. Before we figure out what the fish is going on—let’s see that it works, at least.

>>> @double
... def add(x, y):
...     """Returns the sum of x and y."""
...     return x + y>>> add.__name__
'add'
>>> add.__doc__
'Returns the sum of x and y.'

Hooray! So then, back to business.

Higher-Order Decorators

Before we get back to functools.wraps, we need to talk about higher-order decorators—and it’s best to start with something simpler:

Decorator Factories

At the end of the day, a decorator is just a function, right? And as such, it can be a returned from another function. “Dude this is so meta” aside, it’s not that hard to imagine:

def multiply(m):
    def decorator(f):
        def wrapper(*args, **kwargs):
            return m * f(*args, **kwargs)
        return wrapper
    return decorator

However daunting these inception-esque definitions might seem, they’re actually pretty straightforward: multiply(m) is a decorator factory, which receives m and returns a decorator that, once applied to some function, makes it so its results are multiplied by m. We can then do:

>>> double = multiply(2)>>> @double
... def inc(x):
...     return x + 1>>> inc(x)
4

But also:

>>> triple = multiply(3)>>> @triple
... def inc(x):
...     return x + 1>>> inc(x)
6 # (1 + 1) * 3

It turns out we don’t even have to create the decorator ahead of time; that expression after the @ sign—@me!!!—is actually just that: an expression. We’ve only used simple expressions so far: namely, names. But we can do much more—invocations, arithmetics, even ternary operations! As long as that expression returns a decorator, which can be applied to whatever’s underneath, we’re golden:

>>> @multiply(2)
... def add(x, y):
...     return x + y>>> add(1, 2)
6 # (1 + 2) * 2>>> @multiply(3)
... def add(x, y):
...     return x + y>>> add(1, 2)
9 # (1 + 2) * 3

In this context, you could say multiply is a second-order decorator: a function that returns a function, which receives a function and returns a function. Quite a mouthful—but if you’re catching the drift, it’s pretty much the next logical step.

Functools.wrapsing Stuff Ourselves

Let’s try to simulate functools.wraps—copy over the function name and docstring. Here’s what I got:

def warps(original):
    def fix(wrapper):
        wrapper.__name__ = original.__name__
        wrapper.__doc__ = original.__doc__
        return wrapper
    return fix

And when we apply it…

>>> def double(f):
...     @wraps(f)
...     def wrapper(*args, **kwargs):
...         return 2 * f(*args, **kwargs)
...     return wrapper>>> @double
... def inc(x):
...     return x + 1>>> inc(1)
4
>>> inc.__name__
'inc'

It works! Now, let’s break it down. When we decorated inc with @double, it was as if we did inc = double(inc), so inc was passed into double as f. Then, we had to resolve wraps(f)—which returned the decorator fix, whose purpose is to fix its target to be like original, which is bound to inc. This fix is then used as wrapper’s decorator, so once wrapper is ready, we pass it on for fixing, and get its __name__ and __doc__ replaced by original's—that is, inc's—values. We end up with a wrapper that behaves like the decorated version, but resembles the original one—and that’s what we return to replace inc.

Parametrized Tracing

Second-order decorators are handy for more than just fixing other decorators. For example, in our previous tracing example, we hardcoded the print function in; what if we’d like to decorate our code to log stuff into a file?

Whenever you think “parameterized decorator”—it means you’re going to end up one order up, to accomodate for those parameters, and define your decorator inside a scope that has them fixed to some actual arguments. So if we’d like to parameterize our trace decorator to receive a log function, we’d indent it once and wrap it up in a scope with it:

def trace(log):
    def decorator(f):
        def wrapper(*args, **kwargs):
            log(f'enter {f.__name__}')
            try:
                return f(*args, **kwargs)
            finally:
                log(f'leave {f.__name__}')
        return wrapper
    return decorator

Now we can apply it like so:

>>> @trace(print)
... def div(x, y):
...     return x / y>>> div(4, 2)
enter div
leave div
2.0

Or like so:

>>> def write(message):
...     with open('/tmp/log.txt', 'a') as fp:
...         fp.write(message + '\n')>>> @trace(write)
... def div(x, y)
...     return x / y>>> div(4, 2)
2.0
>>> print(open('/tmp/log.txt').read())
enter div
leave div

Semi-parametrization

If you’re still here, you’re either a jedi, or super high—in any case, let’s take it one step further. Some motivation: most of the time, our log is going to be the print function, so let’s bake it in as a default argument:

def trace(log=print):
    ...

However, this has the unfortunate side effect that the second-order decorator has to be invoked (albeit with no arguments) in order to produce the regular, first-order decorator that is applied to the function:

@trace()
def div(x, y):
    return x / y

Notice the parenthesis? It’s pretty annoying—and if you forget them, you’re in for some cryptic error messages. What I’d like to do is create a… creature, an abomination, which doubles as both a first-order decorator and a second-order decorator. Ready?

def trace(f=None, /, *, log=print):
    if f is None:
        return lambda f: trace(f, log=log)
    def wrapper(*args, **kwargs):
        log(f'enter {f.__name__}')
        try:
            return f(*args, **kwargs)
        finally:
            log(f'leave {f.__name__}')
    return wrapper

OK, let’s break it down. When we call the decorator like so:

@trace(log=write)
def div(x, y):
    return x / y

What happens is that write is passed in as the keyword-only parameter log, and the position-only parameter f remains None; this in turn makes the if statement come true, and returns a lambda that can be thought of as a proxy-decorator. This lambda is then applied to div, but what it actually does is just re-invoke trace, this time with its arguments complete. The rest is pretty much the same as before: we define a wrapper in a scope with f and log, and get ourselves a traced division function! But what happens if…

@trace
def div(x, y):
    return x / y

In this case, div is passed immediately into trace, and log defaults to print. We skip the if statement altogether, and just define a wrapper—nothing to see, move along.

Conclusion

This has been fun! Functions, and the so-called functional programming style, which treats them as first-class citizens and juggles them around, really wrinkles your brain—but it’s really powerful, and once you get it, surprisingly straightforward. Stay with me as we venture deeper into functions, this time taking them apart and tinkering with them under the hood.

The Advanced Python Programming series includes the following articles: