Python: Meta-Programming
Background
Metaprogramming is a programming technique in which computer programs have the ability to treat other programs as their data. It means that a program can be designed to read, generate, analyze or transform other programs, and even modify itself while running. — Wikipedia
Meta-programming in simpler terms is writing some code that can change the fundamental behavior of a language. You might be doing this without realizing. In python there are three common ways to write code that alters the behavior of existing code. You may have even used either of these techniques.
- Function Decorating
- Class Decorating
- Metaclasses
Function Decorating
Experienced python programmers will tend to use Functional Decorators to enhance the function with pre/post/during actions. This technique can cut down on boilerplate code that might be very repetitive or function calls that soak up the body of the function.
In a made up scenario of wanting to time how long a function call takes we make have a function decorator that looks like the following:
from functools import wraps
import timedef timeit(f):
@wraps(f)def wrapper(*args, **kwargs):
start = time.time()
resp = f(*args, **kwargs)
end = time.time()
return (resp, end-start)
return wrapper
@timeit
def hello_world():
time.sleep(1)
return "Hello world"
The function hello_world is being decorated. The function timeit is called the decorator. This is a very rudimentary example but you can learn a lot from this.
The @ symbol initiates the decoration. This is applying the decorator (timeit) to the function hello_world. The other segment to note is call to functool’s @wraps. But before we move to that lets quickly write some driver code to utilize the decorated function and verify the behavior.
print(hello_world())
resp, duration = hello_world()
print("Function: "+str(hello_world.__name__))
print("Resp: "+str(resp))
print("Duration: "+str(duration))# OUTPUT
# ('Hello world', 1.000917911529541)
# Function: hello_world
# Resp: Hello world
# Duration: 1.00130796432
As you can see the above code for function declaration hello_world only returns one string value. But after wrapping hello_world with the decorator function it now produces a tuple return statement of (string, duration) produced by the timeit decorator. COOL HUH!
Just as a quick aside we can also apply decorators by doing:
decorated_hello_world = timeit(hello_world)
print(decorated_hello_world())
# OUTPUT
# ('Hello world', 1.000917911529541)Now back to a very important line of code the @wraps(f). Lets look at a scenario where that does not exist.
def timeit2(f):def wrapper(*args, **kwargs):
start = time.time()
resp = f(*args, **kwargs)
end = time.time()
return (resp, end-start)
return wrapper@timeit2
def hello_world2():
time.sleep(1)
return "Hello world"
After getting rid of wraps and lets run our driver code again.
print(hello_world2())
resp, duration = hello_world2()
print(hello_world2.__name__)
print(resp)
print(duration)# OUTPUT
# ('Hello world', 1.0033891201019287)
# wrapper
# Hello world
# 1.00364780426
As you can see hello_world2.__name__ value should be the decorated function hello_world2 rather than wrapper. So lets try to understand what happened here, the wrapper function timeit2 replaced the original function hello_world2 and returned the function wrapper instead. If we dig into the wraps in the functools.py module we will see the following.
WRAPPER_ASSIGNMENTS = ('__module__', '__name__', '__qualname__', '__doc__',
'__annotations__')
def wraps(wrapped,
assigned = WRAPPER_ASSIGNMENTS,
updated = WRAPPER_UPDATES):"""Decorator factory to apply update_wrapper() to a wrapper function
Returns a decorator that invokes update_wrapper() with the decorated
function as the wrapper argument and the arguments to wraps() as the
remaining arguments. Default arguments are as for update_wrapper().
This is a convenience function to simplify applying partial() to
update_wrapper().
"""return partial(update_wrapper, wrapped=wrapped,
assigned=assigned, updated=updated)
assigned=assigned, updated=updated)
We can notice that the __module__, __name__, __qualname__, __doc__ and __annotations__ attributes will get carried along to the new function when using the wraps decorator.
I would highly recommend to use @wraps(f) annotation to your decorators to not notice bizarre behaviors when you are trying retrieve function attributes such as function docs or name.
Examples of function decorators
Two of my favorite frameworks that use plenty of function decorators to reduce boilerplate code are:
- Flask (lightweight web framework): Decorators are used for routes, views, redirects, login, and plenty of more scenarios.
- Click (framework for building flexible CLIs): Decorators are used for labeling commands, hierarchy, and other misc features that you would want in a CLI
Class Decorating
Class decorators are a bit more complicated, as there is no builtin function to functools.wraps function to “wrap” classes and pass the class internal functions up to the wrapper classes.
Lets look at decorating a class with a class definition.
def decorate_class(cls):class ClsWrapper(object): def __init__(self, *args, **kwargs):
self.wrapped = cls(*args, **kwargs)
setattr(self.wrapped, 'one_plus_one', lambda: 1 + 1) def __getattr__(self, name):
return getattr(self.wrapped, name) return ClsWrapper
@decorate_class
class Foo:
pass
We are decorating the class Foo with the decorator decorate_class. This is an example taking advantage of the python class data model (internals of defining a class). The __getattr__ is invoked whenever a class function invocation is attempted but the function called does not exist. The __init__ function of ClsWrapper creates the instance of the wrapped class but also calls setattr to add a new attribute to the wrapped class called one_plus_one.
Lets look at the following driver code:
foo = Foo()
print(foo.one_plus_one())# Output:
# 2print(type(foo))# Output
# <class '__main__.decorate_class.<locals>.ClsWrapper'>
The output is as expected and the process call is as following:
- When loading the module the decorator will get evaluated and the class Foo will get swapped with ClsWrapper.
- Upon constructing Foo.__init__ it will actually invoke an instance of ClsWrapper.__init__ (noted by example when printing type(foo))
- Now the __init__ function of ClsWrapper then calls the Foo.__init__ function as well as the setattr function which adds a new function called one_plus_one.
- When calling one_plus_one function it will go to ClsWrapper, which then says I do not have a function called one_plus_one and redirect the call to __getattr__.
- __getattr__ being invoked will then attempt to fetch the function definition from self.wrapped (via getattr) which is an instance of Foo which was modified by setattr.
The process describes the reason why foo.one_plus_one() can actually resolve even though Foo does not have that as a function member.
Class decorators are much less common as it is much harder to debug and trace issues when dynamically manipulating the attributes of the class via decorators. Inheritance also behaves in a funky way when using class level decorators.
Examples of class decorating
Unfortunately I can not seem to find good examples of where you may want to do this. Most frameworks that I commonly use or frequent tend to use mixin patterns, functiondecorating or metaclasses to solve problems rather than decorating classes.
Metaclasses
We made it close to the end, and as you can see things are starting to get more confusing. Meta-classes are all the way at the end because it inherently changes the behavior of a class itself. I highly recommend that you understand how they work but avoid using them unless necessary in practice as it can increase complexity of your project as well as make weird class behaviors that can be difficult to debug.
Before going to meta-classes lets look at the data model of python classes.
The meta-class can determine the behavior of the instantiation of a class all the way to what internal capabilities that a class will be available. The other key point to note is that the meta-class is inherited to all subclasses. Instead of reading more about it lets look at an example of meta-classes using our timeit decorator.
We will be creating a scenario where all functions of a given class will automatically be decorated with a function called timeit.
import types
import time
from functools import wraps
def timeit(f):
@wraps(f)def wrapper(*args, **kwargs):
start = time.time()
resp = f(*args, **kwargs)
end = time.time()
return (resp, end - start) return wrapper
# A metaclass that replaces methods of its classes
class TimeMeta(type): def __new__(cls, name, bases, attr):# Replace each function with a decorated version of the functionfor name, value in attr.items():
if type(value) is types.FunctionType or type(value) is types.MethodType:
attr[name] = timeit(value)
# Return a new type called TimeMetareturn super(TimeMeta, cls).__new__(cls, name, bases, attr)
# Test the metaclass
class Animal(metaclass=TimeMeta): def talk(self):
time.sleep(1)
print("Animal talk")
class Cow(Animal): def talk(self):
time.sleep(1)
print("Moo")
Notice that we created a new meta-class called TimeMeta which inherits from type. It implements the function __new__ which is called when a request for an instance of a class is made. Inside the __new__ function we went in and swapped out all the attributes that are of type Function or Method and decorated them with the timeit function. This all occurs at the instantiation request.
The interesting part about the meta-class is that it is inherited and it is not a decoration but a different way to create a class instance.
So lets look at the following driver code:
animal = Animal()
cow = Cow()print(type(animal))
print(type(cow))
print(animal.talk())
print(cow.talk())# Output
# <class '__main__.Animal'>
# <class '__main__.Cow'>
# Animal talk
# (None, 1.0034050941467285)
# Moo
# (None, 1.004560947418213)
Unlike before there is no issues with the type function as it points to the appropriate class instance. But notice that the Cow class which is a subclass of Animal also inherits the meta-class automatically and also applies the class __new__ logic to itself.
So unlike multiple inheritance and mixins we can only have one meta-class. Lets try it with another popular meta-class ABCMeta. It is from a library called abstract base class which is for creating abstract classes and abstract methods.
from abc import ABCMetaclass Cow(Animal, metaclass=ABCMeta): def talk(self):
time.sleep(1)
print("Moo")
You will receive an error which says:
TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its basesSo the meta-class has to follow one hierarchy and you cant mix and match meta-classes.
Again I will reiterate meta-classes are important to understand but avoid using them if you can. Keep it simple! Do not increase complexity because the side effects of a poorly written meta-class is applied to all objects in the inheritance chain. Can make the issues hard to identify.
Examples of meta-classes
The best example of writing a meta-class is in the standard python library in abc.py which stands for Abstract Base Class. This is how you can create abstract classes in python. It is the most commonly used meta-class along with its function decorator @abstractmethod.
Conclusion
Meta-programming provides a great way to declare certain behavior of your functions while cutting down on repetitive boilerplate code. Most of the examples here show examples of adding new behavior but you can very easily remove or modify existing behavior of your software using these meta programming techniques.
Before going too far down the rabbit hole understand your codebase first. See if meta programming can improve readability and testability of your code.
Keep in mind to not create a solution in search of a problem but find the problem first and solve it effectively with the right solution.
