Decorators in Python: Why and How to Use Them and Write Your Own

Level up your ability to follow SOLID principles with an intermediate-level Python technique.

Image credit: https://www.wocintechchat.com/blog/wocintechphotos

When you’re just getting started with Python, essentially everything you’re learning is foundational. There is not a lot of time or motivation to stop and ask when will I use this? when you’re learning about basic things like data types, conditionals, loops, and functions…because the answer is as a Python programmer, you’ll use all of these things all the time! Whether you are a data scientist building machine learning models, or a software developer building the backend of a website, it’s unlikely you’ll be able to make any progress at all if you don’t know the basics.

But once you understand the essential foundations of Python, it can be challenging to identify which new tools and concepts will be helpful — and possibly even necessary—in your advancement, compared to those that might just be a waste of your time.

In this story, I’ll walk you through:

1. Review of SOLID software development principles
2. Examples of “tacked-on” scenarios that are challenging to implement while following the SOLID principles
3. How a decorator approach overcomes this challenge
4. Using Python decorators written by others
5. Writing your own decorators in Python

The code in this blog post can also be found in this Google Colaboratory notebook:

2020_08_20_Python_Decorators_Demo.ipynb

SOLID principles

If you’re planning on contributing to “real” software projects, not just one-off scripts or personal projects, it’s a good idea to be familiar with the SOLID principles.

Uncle Bob Consulting, LLC

The SOLID principles were originally created by Robert C. Martin, AKA “Uncle Bob” of Uncle Bob Consulting. To quote from his website:

The principles are mental cubby-holes. They give a name to a concept so that you can talk and reason about that concept. They provide a place to hang the feelings we have about good and bad code. They attempt to categorize those feelings into concrete advice.

The 5 SOLID principles are:

• Single-Responsibility Principle
• Open-Closed Principle
• Liskov Substitution Principle
• Interface Segregation Principle
• Dependency Inversion Principle

The two we’re going to focus on in this blog post are the single-responsibility principle and the open-closed principle. These are the two principles that are most challenging to follow in the “tacked-on” scenarios described later.

Single-responsibility principle

Every component of a piece of software should only be responsible for a single set of functionality. In the context of the examples in this blog, a “component” will mean a Python function.

Open-closed principle

Already-written software should be “closed” for modification, but “open” for extension. Meaning, we should avoid re-writing existing code whenever possible, but ideally we should be able to add extended functionality to that code.

“Tacked-on” software scenarios

While “SOLID” is a widely-known term in software development, “tacked-on” is just a name that I came up with for a category of software scenarios where you have a perfectly-functional software library, but now a stakeholder is asking you to “tack on” some more functionality that is related to the original functionality while simultaneously being conceptually distinct.

Some examples of “tacked-on” software scenarios are:

• Caching
• Logging
• Access control
• Input validation
• Tweaks to input or output format

In all of these cases, you want to have the original functionality, but also every time you invoke that original functionality, you want something else to happen on top of it.

Caching example

Let’s focus on caching as a specific example of a “tacked-on” scenario. In general, caching means that you save the result of a costly operation in a cache (usually implemented as some form of a dictionary/hashmap) so that later you can just look up that result instead of re-doing that costly operation.

If you’ve done any technical interview algorithms practice before, you’ve likely encountered the task of optimizing the recursive Fibonacci number algorithm. We’ll use it as a stand-in for a more realistic computing task. An example implementation of this algorithm (found in the Python docs) is:

def fib(n):
    if n < 2:
        return n
    return fib(n-1) + fib(n-2)

Let’s assume that your coworker already wrote that code as part of a larger software project, as well as this code (standing in for a more realistic unit test suite) to test the functionality:

assert fib(25) == 75025

The problem with this implementation is that you end up re-computing the same values repeatedly. For example, if you’re computing fib(10) that means fib(9) + fib(8). Then fib(9) =fib(8) + fib(7) and fib(8) = fib(7) + fib(6). Already you can see that fib(8) and fib(7) are being computed twice, and this repetition cascades through the rest of the recursion. How can you make this code faster?

The “known” way to optimize your Fibonacci solution is to add in caching, so that once fib(8) is calculated, you never have to calculate it again, you just have to retrieve it from the cache. But how to do that while following the SOLID principles?

Sub-optimal caching solutions

The most obvious/common way to address this is to make a cache outside of the function, then adapt the function to check the cache before doing the computation and add all computed values to the cache. Something like this:

cache = {}
def fib(n):
    if n < 2:
        return n
    if n in cache:
        return cache[n]
    result = fib(n-1) + fib(n-2)
    cache[n] = result
    return result

The test suite will still pass ✅

assert fib(25) == 75025

But what about the SOLID principles?

Single-responsibility ❌

Now instead of a function that computes Fibonacci numbers, we have a function that computes Fibonacci numbers and manages adding and retrieving things from a cache.

Open-closed ❌

We did not leave the existing function “closed”. If you looked at the Git blame for the function, 3 of the original lines of the function would be the same, 1 would be removed, and 6 would be added. You’ve re-written the majority of this whole function, and if you later decide to use a different kind of cache, you would need to “open up” the function again.

Another popular implementation of this would be to avoid creating a global cache by converting the function into a method of a class, and making the cache a member variable of that class. This has the same single-responsibility ❌ and open-closed ❌ issues as the previous solution, and additionally would require re-writing of the unit test ❌ (and all code that currently expects this to be a standalone function rather than a class method).

The decorator approach

The decorator approach is a great way to solve this kind of problem. It goes back to the classic “Gang of Four” Design Patterns book published in 1994.

Design Patterns: Elements of Reusable Object-Oriented Software

While that book describes “composition over inheritance” in the context of designing OOP classes, we can translate it in this context to be about the composition of functions.

In general, the decorator approach means you want to “wrap” your code in this tacked-on functionality, meaning you create something that is composed of the original functionality, now decorated with the tacked-on functionality.

Because we don’t need to modify the existing code, by using a decorator approach we are able to “tack on” the new functionality while still following the single-responsibility and open-closed SOLID principles.

Clarification

There are many resources describing the Decorator Pattern, like this article. This terminology is specific to a narrow type of object-oriented “decorator classes”, which can theoretically be implemented using Python decorators. Decorator classes are addressing the same kinds of scenarios we have been discussing, and the general idea is similar. But “decorators in Python” refers to a more high-level construct that can apply to both functions and classes. This potential confusion around the naming of Python decorators is addressed but not resolved in the Python Enhancement Proposal from 2003.

Using python decorators

Before we dive into writing our own custom Python decorator, let’s use one that has already been written for us. Conceptually this is similar to learning how to use existing functions like print() or len() before attempting to write functions for ourselves.

We’ll return to the Fibonacci numbers example from before:

def fib(n):
    if n < 2:
        return n
    return fib(n-1) + fib(n-2)

Now let’s add an LRU cache from the built-in functools module (built-in meaning we don’t need to install any libraries beyond base Python, we just need to import it):

import functools
@functools.lru_cache(maxsize=None)
def fib(n):
    if n < 2:
        return n
    return fib(n-1) + fib(n-2)

…and that’s it! 🤯

The test suite still passes ✅

Single-responsibility principle ✅

The fib function, as written, is still just a Fibonacci number function. It is not a function combining Fibonacci numbers and caching.

Open-closed principle ✅

We left the original function “closed”. If someone went to look at the Git blame, it would still be fully attributed to the original author. But at the same time, we “opened” it up for extension, since now it’s using caching.

Reality check: In case you are wondering if adding these two lines of code really “did anything”, I ran these two snippets using the %%timeit magic command and fib(25). The original code took 30.9 ms (30.9 milliseconds), and the code with caching took 6 µs (6 microseconds). There are 1000 microseconds in a millisecond, meaning the performance of the code with caching was over 1000x faster than the original—pretty impressive for an addition of 2 lines of code!

So…what just happened?

Mechanically, we just imported a module (functools), then added an @ symbol plus a function from within that module.

Conceptually, we “wrapped” the existing fib function in a cache decorator:

1. The lru_cache decorator took in our function as an argument
2. It added caching functionality
3. It returned a new function composed of fib plus the caching logic
4. Finally, it re-assigned the name fib to the new function, so we could continue using the original interface

Using already-written Decorators

Unfortunately, the Python developers do not maintain a global list of decorators available in base Python, but this GitHub repo contains a pretty good list of those decorators as well as decorators “in the wild” (i.e. located in libraries outside of Python itself):

lord63/awesome-python-decorator

(This repo has not been updated since 2017, let me know in the comments if you have found a better or complementary resource!)

Reality check: If you’re curious how often and in what context you’ll encounter decorators “in the real world”…in my work in software engineering and data science, I most frequently encounter decorators in a backend/full-stack web development context, especially Flask and Dash. Those frameworks use decorators to allow you to write functions that will be wrapped in the full web server functionality. It is essentially impossible to create an application using either of those frameworks without using decorators. Using a Flask decorator looks something like this (the “minimal” application from the Quickstart guide):

from flask import Flask
app = Flask(__name__)

@app.route('/')
def hello_world():
    return 'Hello, World!'

Using decorators that someone else has written is great! You can get some excellent functionality right “out of the box”: just import (if needed), add the @ symbol syntax, and your function is now decorated with added functionality. There are excellent caching, logging, and access control decorators already out there. It can be acceptable to leave decorators as an unopened “black box” if someone has already written a decorator that accomplishes what you need to accomplish!

But sometimes, you might want to use this technique in a context where nobody has written a decorator that does the thing you want to do. So let’s go ahead and explore how you might write a custom decorator.

Python functions as “first-class” objects

Before we can jump right in to writing custom decorators, we need to review how Python functions are “first-class” objects. This is a concept that will be very familiar if you’ve used JavaScript for front-end web development (think callbacks and event listeners) and very foreign if you’ve only used a language like Ruby or Java before, that doesn’t treat functions this way.

Functions in Python:

• Can be stored in data structures
• Can be assigned to variables
• Can be passed to other functions
• Can be nested
• Can capture local state

(Check out this tutorial for more explanation and examples of these properties.)

We’re going to focus on those three bolded properties, since they are key for understanding how decorators work.

Assigning functions to variables, and passing functions to other functions

Here are some short/trivial examples to show what these properties mean. First, let’s set up some functions:

def squared(num):
    return num ** 2
def cubed(num):
    return num ** 3
print(squared(5)) # prints 25
print(cubed(5))   # prints 125

Pretty straightforward. We have two functions, each of which takes a single argument num and returns a polynomial transformation of num. squared squares it, and cubed cubes it.

Now to demonstrate the “first-class” functionality:

def func_plus_two(polynomial_func, num):
    return polynomial_func(num) + 2
first_func = squared
second_func = cubed
print(func_plus_two(first_func, 5))  # prints 27
print(func_plus_two(second_func, 5)) # prints 127

So, we are assigning functions to variables, in this case assigning squared to first_func and cubed to second_func. (There is no particularly useful reason for doing this, it’s just for demonstration purposes.)

Then, maybe more interestingly, we are passing those functions to another function, first passing first_func then second_func into func_plus_two. That function calls the function that was passed into it, then returns the result of the original function + 2. So, instead of 25 and 125, we get 27 and 127, when num = 5. This structure (passing a function into another function) is also called a higher-order function, i.e. we could say “func_plus_two is a higher-order function”.

Nested functions

It’s even more of a “stretch” to connect this example to something realistically useful, but bear with me, it becomes important when we actually get to writing decorators!

Consider this example of nested functions:

def nth_degree(num, n):
    def squared_inner(num):
        return num ** 2
    def cubed_inner(num):
        return num ** 3
    if n == 2:
        return squared_inner(num)
    elif n == 3:
        return cubed_inner(num)

Now instead of having squared and cubed as functions in the global scope, we have functions squared_inner and cubed_inner that only exist within the scope of the nth_degree function. In other words, if we tried to run squared_inner(5) in the global scope, it wouldn’t return 25, it would throw a NameError: name 'squared_inner' is not defined. To invoke (i.e. call) this function, we could use code like:

nth_degree(4, 2) # 4 squared, prints 16
nth_degree(2, 3) # 2 cubed, prints 8

In the first example, we passed in an n of 2, so nth_degree called the squared_inner function on 4 and returned 16. In the second example, we passed in an n of 3, so nth_degree called the cubed_inner function on 2 and returned 8. In reality we could easily re-write this code to avoid inner functions altogether, but I hope you understand the main takeaway that a function can have another function nested inside itself, then return something based on that nested function.

Writing our own comma-adding decorator

Ok, now that we’ve completed the review of first-class Python functions, let’s write a ⭐ decorator ⭐️ with custom functionality!

Let’s say your boss comes to you with this task:

“Our users are having a hard time reading these large numbers. At a glance, is 1000000 1 million or 10 million? Let’s take out the guesswork and put in some commas, like they’re used to in Comma Style format in Excel. So they see something like 1,000,000 instead of 1000000.”

(This is inspired by a real task I had to do as a software developer, back when I worked in software consulting at Crowe. Accounting folks love their commas!)

You are maintaining a library with 5 functions, all of which currently returns a string representation of an integer to be displayed in the user interface. You need to update the output format of those functions while continuing to follow the single-responsibility and open-close principles.

Since you work in a test-driven company, the software developer in test has already started the project by adapting the unit tests to match the new requirements.

Old unit tests:

assert one_million() == "1000000"
assert one_billion() == "1000000000"
assert times_100(7) == "700"
assert minus_10000(150000) == "140000"
assert multiply(300, 800) == "240000"

New unit tests:

assert one_million() == "1,000,000"
assert one_billion() == "1,000,000,000"
assert times_100(7) == "700"
assert minus_10000(150000) == "140,000"
assert multiply(300, 800) == "240,000"

The decorator you decide to write has this overall structure:

def add_commas(func):
    def add_commas_wrapper():
        # call original function
        # add in the commas
        # return the result
    return add_commas_wrapper

In other words, it is a wrapper function that takes in the original function as an argument, then returns an inner function that will call the original function and also add in the commas.

Simplest version of decorator (no arguments)

Rather than trying to write a decorator with complete functionality all at once, let’s start with something very basic, which only handles the library functions without any arguments.

def add_commas(func):
    def add_commas_wrapper():
        original_string = func()
        # needs to be int for string formatting
        original_int = int(original_string)
        # we are ignoring locale, using default thousands sep
        return f'{original_int:,}'
    return add_commas_wrapper

Now that we have that function, we can modify the existing one_million and one_billion functions:

def one_million():
    return "1000000"
def one_billion():
    return "1000000000"
one_million = add_commas(one_million)
one_billion = add_commas(one_billion)

Now we are passing two of the unit tests:

assert one_million() == "1,000,000" ✅
assert one_billion() == "1,000,000,000" ✅
assert times_100(7) == "700" ✅ # "accidentally" passing, <1000
assert minus_10000(150000) == "140,000" ❌ # returns "140000"
assert multiply(300, 800) == "240,000" ❌ # returns "240000"

Adding the syntactic sugar

You might be wondering…what about that @ symbol we were using earlier? How is this a “decorator” like functools.lru_cache if there’s no @ symbol? Well, the @ symbol here is a form of syntactic sugar.

Syntactic sugar is a concept that extends beyond Python specifically. It means some kind of syntax that shortens a common bit of code, usually to make it easier to read and/or harder to mess up.

Let’s revise the previous code snippet to use decorator syntactic sugar:

@add_commas
def one_million():
    return "1000000"
@add_commas
def one_billion():
    return "1000000000"

It’s the same number of lines of code, and it’s actually doing the exact same thing as the previous snippet. But it’s a little cleaner (with fewer parentheses and repetitions of the name of the function), and someone looking at the function definitions (lines starting with def) will be immediately aware that some decoration is happening beyond what they can see in the function, in case later they’re trying to debug the function. In the previous syntax, the decoration is separated from the definition in a way that could end up being confusing, especially with more or longer functions than these trivial examples.

Accommodating arguments

What if we just try to add this decorator to the times_100 function?

@add_commas
def times_100(num):
    return str(num * 100)
print(times_100(7))

😱 what happened? Well, if we look back at the definition of add_commas_wrapper, it’s:

def add_commas(func):
    def add_commas_wrapper():
        original_string = func()
        # needs to be int for string formatting
        original_int = int(original_string)
        # we are ignoring locale, using default thousands sep
        return f'{original_int:,}'
    return add_commas_wrapper

Just like the error message said, that function takes 0 arguments. How can we adapt it to work with the num argument of times_100 while not breaking the functionality of one_million and one_billion, which don’t take any arguments?

We could add some kind of custom logic, like the if statements in the nth_degree nested function, but that means we might need to repeatedly modify add_commas to handle different numbers of parameters. We shouldn’t need to do that, since really all this function needs to do is start with a string and put commas in that string…it shouldn’t matter how many arguments were passed in to the original function.

Fortunately there is a technique to accept an arbitrary number of arguments (i.e. variable-length arguments) in Python! The traditional syntax for this is *args, **kwargs, meaning 0 or more positional arguments (args) then 0 or more keyword arguments (kwargs). So that would work for one_million (0 arguments of any kind), times_100 (1 positional argument), multiply (2 positional arguments), and any other number of positional or keyword arguments.

Let’s add variable-length arguments:

def add_commas(func):
    def add_commas_wrapper(*args, **kwargs):
        original_string = func(*args, **kwargs)
        # needs to be int for string formatting
        original_int = int(original_string)
        # we are ignoring locale, using default thousands sep
        return f'{original_int:,}'
    return add_commas_wrapper

And add the decorator syntax to all the functions:

@add_commas
def one_million():
    return "1000000"
@add_commas
def one_billion():
    return "1000000000"
@add_commas
def times_100(num):
    return str(num * 100)
@add_commas
def minus_10000(num):
    return str(num - 10000)
@add_commas
def multiply(num1, num2):
    return str(num1 * num2)

Now we are passing all tests!

assert one_million() == "1,000,000" ✅
assert one_billion() == "1,000,000,000" ✅
assert times_100(7) == "700" ✅
assert minus_10000(150000) == "140,000" ✅
assert multiply(300, 800) == "240,000" ✅

🎉 🎉 🎉 🎉 🎉

One more thing

If you want to use decorators in production, there’s one more thing you’ll want to include, although it’s irrelevant for passing these particular tests. You might have noticed with the error message earlier that it reported to be from add_commas_wrapper, not times_100. In that example, it wasn’t particularly relevant, but we could imagine another scenario when someone tries to use a float rather than an int, or does something else that breaks the logic inside of add_commas, when we would want to be able to introspect and see the original function causing the error, not the wrapper.

You can find more details about what’s happening and how to address it manually in this tutorial, but luckily there is yet again a function inside of the built-in functools module that can handle this for us! It is called wraps, and the documentation can be found here.

The final version of our code with wraps added is:

import functools
def add_commas(func):
    @functools.wraps
    def add_commas_wrapper(*args, **kwargs):
        original_string = func(*args, **kwargs)
        # needs to be int for string formatting
        original_int = int(original_string)
        # we are ignoring locale, using default thousands sep
        return f'{original_int:,}'
    return add_commas_wrapper
@add_commas
def one_million():
    return "1000000"
@add_commas
def one_billion():
    return "1000000000"
@add_commas
def times_100(num):
    return str(num * 100)
@add_commas
def minus_10000(num):
    return str(num - 10000)
@add_commas
def multiply(num1, num2):
    return str(num1 * num2)

Recap

Reviewing our updated code

The test suite passes, meaning we have successfully implemented the specified “tacked-on” functionality ✅

Single-responsibility principle ✅

Our computational functions are still just doing computation (standing in for the more-sophisticated computations or queries you would do in a real application). We also have a wrapper function that adjusts the output format of the functions, and it’s not concerned with the number of arguments in the original function, it just always converts a string representation of an integer into a format with commas.

Open-closed principle ✅

We did not make any changes to the contents or the call signature of the original functions. But, by adding 15 lines of additional code, we were able to “open” the code for extension to meet the new stakeholder requirements.

Reviewing what we learned

Some kinds of scenarios, which I call “tacked-on” scenarios, can be challenging to implement without violating the SOLID principles, particularly the single-responsibility principle and the open-closed principle. The decorator approach, and Python decorators in particular, are helpful for tackling this challenge.

In some cases, someone has already written a decorator that handles the “tacked-on” scenario you need, especially for common scenarios like caching, logging, and access control. In other cases, you need to write your own decorator for custom functionality.

In order to write your own decorator, you need to recall that Python functions are “first-class” objects, particularly that they can be assigned to variables, can be passed to other functions, and can be nested. With that knowledge in mind, and the help of syntactic sugar and functools.wraps to ensure proper introspection, you can write your own decorators to handle these “tacked-on” tasks while following the SOLID principles.

References

In addition to the references linked above, credit to this excellent tutorial from Dan Bader for helping me formulate the framing:

Python Decorators: A Step-By-Step Introduction - dbader.org

And I’ll link again to this in-depth tutorial from Real Python for anyone who wants to dig into all of the granular details:

Primer on Python Decorators - Real Python

Check out this set of exercises if you want to get some more practice before using this technique in production:

Decorators - Learn Python - Free Interactive Python Tutorial

Thanks for reading, and let me know in the comments if you know of any other notable uses of decorators or additional tutorials!