Higher-Order Functions and Decorators

Published in

Python Supply

8 min readDec 16, 2019

Suppose you are maintaining a module, package, or data structure that has an API with many methods and you want to add the ability to log every API call. Furthermore, you need to enable or disable the logging feature using a single flag. Do you manually add a block of logging code to every method definition? Do you write a wrapper API? How do you ensure the logging feature is modular in its implementation, remains compatible as the API evolves, and is easy to maintain?

Python’s concrete syntax includes support for something called decorators. This is a syntactic feature that acts as a convenient and concise tool for modifying, analyzing, or associating functions and methods at the point at which they are defined. This can reduce redundancy and clutter in code. Importantly, it leverages Python’s native support for a functional programming paradigm to offer you a different kind of modularity and composability that can help you add the logging feature in a relatively quick, concise, and elegant way.

Functions as Values and Arguments

To understand decorators, it is necessary to at least be aware of higher-order functions. In Python, functions can be defined and used just like values. This is a characteristic feature of the functional programming paradigm, which Python supports. Consider the following example in which a function f is defined and then passed as an argument to another function twice. Note that when twice receives the argument f it is assigned to the local variable g. Then this yields g(g(y)) = f(f(y)) = f(y + y) = (y + y) + (y + y). If you assign 2 to y, this will yield g(g(2)) = f(f(2)) = f(2 + 2) = (2 + 2) + (2 + 2) = 8.

>>> def f(x):
...     return x + x
...
>>> def twice(g, y):
...     return g(g(y))
...
>>> twice(f, 2)
8

Just as functions can be arguments, they can also be results. In the example below, a different variant of twice takes a function as its sole input and returns a new function that behaves like its input function but is applied twice.

def f(x):
    return x + xdef twice(g):    # Define a new function locally.
    def h(y):
        return g(g(y))    # Return the local function as the result.
    return h

Because twice(f) is a function, you can apply it to an argument and it will return a result.

>>> twice(f)(2)
8

Decorators and Function Definitions

As a slightly simpler variant of the motivating example described in the introduction, suppose you want to modify some existing functions so that they display their results using print (in addition to returning their results as they normally would). To do this in a reusable way, you can write a higher-order function that takes the original function as an input and returns a new function that also prints the result.

def displays(f):
    
    # This is the new variant of the function `f`.
    def f_displays(x):
        r = f(x)
        print("The result is:", r)
        return r

    return f_displays

This transformer function displays can then be applied to any existing function to give you a new version that also prints its result.

>>> def double(x):
...     return x + x
...
>>> double = displays(double)
>>> double(2)
The result is: 4
4

Python’s concrete syntax lets you do exactly the same thing using a more concise notation: prepend the@ symbol before your higher-order function and place it immediately above the definition of the function you want to transform.

@displays
def triple(x):
    return x + x + x

In the example above, the variable triple (after the decorated definition is executed) refers to the transformed version of the function in the definition. Notice that the result is displayed using print when the function is evaluated.

>>> triple(2)
The result is: 6
6

Because the higher-order function used as a decorator is itself just a function, it can also be the result of a function. Thus, you can create a function that creates decorators! Below, the function displays_with returns a decorator that prints a custom message rather than the hard-coded one in the examples above.

def displays_with(message):

    # Create the function that converts a function
    # into a function that display (i.e., our old
    # decorator).
    def displays(f):

        # This is the new variant of the function `f`.
        def f_displays(x):
            r = f(x)
            print(message, r)
            return r

        return f_displays
    
    # Return the function created above.
    return displays

The decorator syntax allows you to supply the argument to the function that creates a decorator.

@displays_with('The function returned:')
def triple(x):
    return x + x + x

To clarify what is happening, a code block that is functionally equivalent to the code above is presented below.

def triple(x):
    return x + x + x

triple = displays_with('The function returned:')(triple)

Decorators can also be stacked. Suppose you create a decorator that also adds the decorated function to a running list of functions.

functions = []

def function(f):
    functions.append(f)

You can now decorate a function with both decorators.

@displays_with('The function returned:')
@function
def triple(x):
    return x + x + x

Note that order does matter: if you place the function decorator above the displays_with, the function added to the list using function will be the one already modified by displays_with. Thus, when multiple decorators are present they are applied from the bottom up (or, in other words, decorators are right-associative). In the example below, the transformed function triple is added to the list of functions.

>>> functions = []
>>> @function
... @displays_with("The function returned:")
... def triple(x):
...    return x + x + x
...
>>> functions[0](2)
The function returned: 6
6

In the example below, on the other hand, the original version of triple is added to the list of functions.

>>> functions = []
>>> @displays_with("The function returned:")
... @function
... def triple(x):
...    return x + x + x
...
>>> functions[0](2)
6

Decorators and Class Definitions

Just as functions in Python can be used as values, so can classes. This article will not go into much depth on this subject. It is enough to see that the same examples presented involving function definitions have corresponding examples involving class definitions.

First, note that you can define a function that takes a class (not an object of the class, but the class itself) as an input. The function check below takes a class as an input and checks if it has a method called method.

>>> class C:
...    def __init__(self, attr):
...        self.attr = attr
...
>>> class D:
...    def method(self):
...        pass
...
>>> def check(cls):
...    return callable(getattr(cls, 'method', None))
...
>>> check(C), check(D)
(False, True)

Decorators can be added to a class definition in the same way that they can be added to a function definition. In the examples below, the decorator displayable checks whether a class definition includes a method called display and raises an exception if it does not.

def displayable(cls):
    if not callable(getattr(cls, 'display', None)):
        raise Exception('objects of this class are not displayable')@displayable
class C:
    def display(self):
        return 'C'

The expected behavior can be seen in the example below.

>>> try:
...     @displayable
...     class D:
...         def method(self):
...             pass
... except Exception as e:
...     print(e)
...
objects of this class are not displayable

Use Cases

As illustrated above, decorators are a reusable way to analyze, log, associate, or transform a function, method, or class by adding just one line above their definition. The original motivating use case, as well as a few others, are reviewed below.

Adding Logging to an API

Logging is a compelling use case for decorators because it illustrates how they can save significant time and effort. To review: you have a large API and need to log the inputs and outputs of every API call. The simple API below can act as a placeholder or this example.

class Database:
    def __init__(self):
        self.data = []
    
    def insert(self, entry):
        data.append(entry)
        return True

    def find(self, entry):
        return entry in self.data

One approach you can take is to implement a single decorator that you will reuse for every method in the API implementation.

log = []def logged(f):    def logged_f(db, inp):
        outp = f(db, inp)
        log.append({'method':f.__name__, 'in':inp, 'out':outp})
        return outp    return logged_f

You can add the above decorator before every public method definition.

class Database:
    def __init__(self):
        self.data = []    @logged
    def insert(self, entry):
        self.data.append(entry)
        return True    @logged
    def find(self, entry):
        return entry in self.data

Below is what you might see in the log after a few API calls are made.

>>> db = Database()
>>> db.insert('alice')
>>> db.find('alice')
>>> db.find('bob')
>>> log
[{'method': 'insert', 'in': 'alice', 'out': True},
 {'method': 'find', 'in': 'alice', 'out': True},
 {'method': 'find', 'in': 'bob', 'out': False}]

As an exercise, you may want to try writing a single decorator for an entire class definition in order to avoid adding a decorator to every function. You may find the built-in function dir useful for this purpose.

Defining Hooks, Extensions, and Event Handlers

Decorators are used in a number of popular libraries, such as Flask. In the example below drawn from the Flask documentation, Flask is used to set up an HTTP server with a single route. The function that handles processing of a requests and the construction of a response is associated with that event using a decorator provided by the Flask API.

from flask import Flask
app = Flask(__name__)@app.route('/')
def hello_world():
    return 'Hello, World!'

Note that the decorator is itself a method, and also that it is technically higher-order (in that it takes an argument consisting of the route path and returns a decorator that is then applied to the function being defined).

Analyzing or Measuring Functions/Methods

As illustrated in the example with the class decorator displayable, decorators can be used to implement static or dynamic analyses of functions, methods, and classes. A static analysis might only examine its input function/class (or the code inside it) at the time of the definition without actually running the code or modifying the function/class itself. A dynamic analysis might run the code itself or it might modify the code to measure its own operation in some way. Because you have already seen an example of the former, examples of the latter use case are presented below.

For the first example, suppose you want to test that a method always returns positive outputs in a range of inputs. The decorator definition below illustrates one way that this can be accomplished.

def check(f):    # Run some tests on `f`.
    for x in range(-10,11):
        if f(x) < 0:
            raise ValueError('incorrect negative output')    # Do not modify the original function.
    return f@check
def square(x):
    return x * x

For the second example, consider a situation in which you want to check the running time of various methods. You can use decorators in conjunction with the built-in time package.

def timed(f):    from time import time
    def time_f():
        ts = time()
        result = f()
        te = time()
        print(f.__name__ + ":", te-ts, "seconds")
        return result    return time_f

The example below demonstrates how the timed decorator defined above might be used.

>>> @timed
... def work():
...     from time import sleep
...     sleep(1)
...
>>> work()
work: 1.000389575958252 seconds

One issue that might arise when an analyzing functions in this way, especially when you are stacking decorators, is that the decorated function will not preserve the original function’s metadata.

>>> work.__name__
'time_f'

To avoid this, you can use the built-in high-order function wraps (which can itself be used as a decorator) found in the built-in functools library.

def decorated(f):

    from functools import wraps
    @wraps(f)
    def decorated_f():
        return f()

    return decorated_f

Using wraps as in the above example ensures the metadata of the original function is preserved.

>>> @decorated
... def f():
...     pass
...
>>> f.__name__
'f'