Adventures of an iOS developer in Python

A dive into properties, descriptors, decorators, annotations and metaclasses

Introduction

Hi. I attended the EuroPython 2018 conference in Edinburgh 23–29 July 2018. I really liked Francesco Pierfederici’s workshop called Ridiculously Advanced Python and I want to share my experiences with you.

The idea of the workshop was the implementation of type checking using the following Python features:

• properties
• descriptors
• decorators
• annotations
• metaclasses

Francesco Pierfederici’s workshop

Before we begin

This article is focused on junior developers.

Be aware that some examples of this workshop are not best practices and shouldn’t be used in real projects. They are good for the demonstration of Python features.

Please install Python 3 to make sure all code examples work as expected. Personally, I prefer pyenv tool to have multiple Python versions on single machine.

Problem statement

We have the following code snippet, it represents the simplest 2D point:

class Point:
    def __init__(self, x, y):
        self.x = x
        self.y = y
 
    def move_by(self, dx, dy):
        self.x += dx
        self.y += dy
 
    def __str__(self):
        return 'Point: {}'.format(self.__dict__)

Initialiser (method __init__) adds x and y attributes to point instance and method move_by modifies them. It looks clean and simple.

Let’s do a bit of programming:

>>> p = Point(1, 2)
>>> print(p)
Point: {'x': 1, 'y': 2}
>>> p.move_by(1, 2)
>>> print(p)
Point: {'x': 2, 'y': 4}

As we can see, attributes xand ychanged properly. Our class Point works as expected. But the problem is that it’s easy to misuse it. For instance, if we use strings instead of ints, the result is completely different and undesirable:

>>> p = Point('1', '2')
>>> print(p)
Point: {'x': '1', 'y': '2'}
>>> p.move_by('1', '2')
>>> print(p)
Point: {'x': '11', 'y': '22'}

For Python everything looks good, it just concatenates strings. But that’s not what we actually want. We definitely miss type checking. Let’s check if we can use properties to solve this problem.

Properties

If you are not familiar with properties, you can check the Python tutorial. In short, this is a way to hide specific attribute and define setter and getter interfaces to manipulate them.

Let’s improve our Point class. We can turn attributes x and y to properties:

class Point:
    def __init__(self, x, y):
        self.x = x
        self.y = y
 
    @property
    def x(self):
        return self._x
 
    @x.setter
    def x(self, value):
        assert isinstance(value, int), 'Booooo! Expecting an int'
        self._x = value
 
    @property
    def y(self):
        return self._y
 
    @y.setter
    def y(self, value):
        assert isinstance(value, int), 'Booooo! Expecting an int'
        self._y = value
 
    def move_by(self, dx, dy):
        self.x += dx
        self.y += dy
 
    def __str__(self):
        return 'Point: {}'.format(self.__dict__)

Let’s call move_by method with int arguments:

>>> p = Point(1, 2)
>>> print(p)
Point: {'_x': 1, '_y': 2}
>>> p.move_by(1, 2)
>>> print(p)
Point: {'_x': 2, '_y': 4}

Everything works as expected. Output slightly changed because properties x and y manipulate with private attributes _x and _y.

Now let’s pass string arguments:

>>> p = Point('1', '2')
Traceback (most recent call last):
  File "playground.py", line 37, in <module>
    p = Point('1', '2')
  File "playground.py", line 3, in __init__
    self.x = x
  File "playground.py", line 12, in x
    assert isinstance(value, int), 'Booooo! Expecting an int'
AssertionError: Booooo! Expecting an int

Boom! Execution was interrupted. That’s exactly what we are trying to achieve — properties x and y should accept just int arguments.

The actual type checks are done in property setters. Such approach is well-known, it is easy to understand and maintain. But it also has a downside: code duplication. We have to do the same check for each property over and over again. That doesn’t sound good. Let’s go further — write a descriptor.

Descriptors

If you are not familiar with descriptors, please check the documentation. In short, a descriptor is an object attribute with “binding behavior”. It specifies what happens when an attribute is referenced on a model, it allows developers to manage attribute access: set, get and delete.

Let’s do some coding. We have the following descriptors:

class TypeChecker:
    required_type = object
 
    def __init__(self, name):
        self.name = '_{}'.format(name)
 
    def __get__(self, instance, owner=None):
        return instance.__dict__[self.name]
 
    def __set__(self, instance, value):
        assert isinstance(value, self.required_type), \
               'Booooo! Expecting a {}'.format(self.required_type.__name__)
        instance.__dict__[self.name] = value
 
 
class IntType(TypeChecker):
    required_type = int

The TypeChecker defines the getter and setter. The getter just returns an attribute from the instance (point in our case). The setter stores a new value as attribute in the instance and it also does the type check and interrupts execution in case the value has a wrong type.

The TypeChecker’s attribute required_type holds the actual type for type checking. By default, it’s object, so any type is valid. The IntType is a subclass of TypeChecker, it just defines required type. In this case it’s int.

Let’s modify our Point class a bit:

class Point:
    x = IntType('x')
    y = IntType('y')
 
    def __init__(self, x, y):
        self.x = x
        self.y = y
 
    def move_by(self, dx, dy):
        self.x += dx
        self.y += dy
 
    def __str__(self):
        return 'Point: {}'.format(self.__dict__)

As you can see, we added x and y attributes to the Point class itself. These attributes are instances of descriptor IntType. It means that Python calls getter and setter of proper descriptor when we manipulate with x and y attributes of Point instance.

Let’s check if everything works as expected:

>>> p = Point(1, 2)
>>> print(p)
Point: {'_x': 1, '_y': 2}
>>> p.move_by(1, 2)
>>> print(p)
Point: {'_x': 2, '_y': 4}

Yep, nothing changed. And what about string usage?

>>> p = Point('1', '2')
Traceback (most recent call last):
  File "playground.py", line 41, in <module>
    p = Point('1', '2')
  File "playground.py", line 25, in __init__
    self.x = x
  File "playground.py", line 12, in __set__
    'Booooo! Expecting a {}'.format(self.required_type.__name__)
AssertionError: Booooo! Expecting a int

Also no changes. Descriptor implements type checking and interrupts execution in case of non-int argument.

It looks like extra work to create own descriptors, but it will save coding efforts in the future: you don’t have to copy-paste type checking logic — you can just reuse type checker descriptors. Code readability and testability also increase.

But one thing is annoying — you have to repeat attribute name twice. E.g. x in x = IntType(‘x’).

Let’s check if we can improve our solution by using decorators.

Decorators

If you are not familiar with decorators, you can check Python tutorial. In short, decorator allows to add any functionality to any object. In most cases that object is a function but in our case it’s a class.

Let’s extend code from previous section — add decorator type_check and decorate our Point class:

class TypeChecker:
    required_type = object
 
    def __init__(self, name=None):
        self.name = '_{}'.format(name)
 
    def __get__(self, instance, owner=None):
        return instance.__dict__[self.name]
 
    def __set__(self, instance, value):
        assert isinstance(value, self.required_type), \
               'Booooo! Expecting a {}'.format(self.required_type.__name__)
        instance.__dict__[self.name] = value
 
 
class IntType(TypeChecker):
    required_type = int
 
 
def type_check(cls):
    for var_name, checker in cls.__dict__.items():
        if isinstance(checker, TypeChecker):
            checker.name = '_{}'.format(var_name)
    return cls
 
 
@type_check
class Point:
    x = IntType()
    y = IntType()
 
    def __init__(self, x, y):
        self.x = x
        self.y = y
 
    def move_by(self, dx, dy):
        self.x += dx
        self.y += dy
 
    def __str__(self):
        return 'Point: {}'.format(self.__dict__)

Definition of attributes x and y is slightly changed — the name argument is omitted. Why so? The answer lays inside decorator type_check. It receives our Point class object as an argument, iterates through the type checker attributes and sets up their name attribute.

Let’s see if we can still create our point with int arguments:

>>> p = Point(1, 2)
>>> print(p)
Point: {'_x': 1, '_y': 2}
>>> p.move_by(1, 2)
>>> print(p)
Point: {'_x': 2, '_y': 4}

Works as it used to work. And string arguments?

>>> p = Point('1', '2')
Traceback (most recent call last):
  File "playground.py", line 49, in <module>
    p = Point('1', '2')
  File "playground.py", line 33, in __init__
    self.x = x
  File "playground.py", line 12, in __set__
    'Booooo! Expecting a {}'.format(self.required_type.__name__)
AssertionError: Booooo! Expecting a int

Nice, interruption occurred. It means that type checking is still there.

So, we simplified the definition of the x and y attributes. Our code looks a bit cleaner. And this logic is reusable, we just have to reuse our type_check decorator. But can we go further? Yes, we can. We just need to use annotations.

Annotations

If you are not familiar with annotations, you can check PEP 526. In short, annotations are specially-formatted comments that are associated with variables and specific parts of functions. Annotations allow to increase code readability but have no impact to execution at run time. Annotations could be used for documentation generation and code analysis.

We can reduce our code by using annotations:

class TypeChecker:
    required_type = object
 
    def __init__(self, name=None):
        self.name = '_{}'.format(name)
 
    def __get__(self, instance, owner=None):
        return instance.__dict__[self.name]
 
    def __set__(self, instance, value):
        assert isinstance(value, self.required_type), \
               'Booooo! Expecting a {}'.format(self.required_type.__name__)
        instance.__dict__[self.name] = value
 
 
def type_check(cls):
    for var_name, var_type in cls.__annotations__.items():
        class Checker(TypeChecker):
            required_type = var_type
 
        setattr(cls, var_name, Checker(var_name))
    return cls
 
 
@type_check
class Point:
    x: int
    y: int
 
    def __init__(self, x, y):
        self.x = x
        self.y = y
 
    def move_by(self, dx, dy):
        self.x += dx
        self.y += dy
 
    def __str__(self):
        return 'Point: {}'.format(self.__dict__)

As you maybe already noticed, we have removed IntType, modified the type_check decorator a bit and turned the x and y attributes to annotations.

Before we continue, it’s worth to mention that we can get a list of class annotations by calling __annotations__:

>>> print(Point.__annotations__)
{'x': <class 'int'>, 'y': <class 'int'>}

But what happens if we call __dict__? Let’s figure it out:

>>> print(Point.__dict__)
{'__module__': '__main__', '__annotations__': {'x': <class 'int'>, 'y': <class 'int'>}...

As you can see, all annotations are accessible via separate key __annotations__. If you try to access annotation x directly as an attribute, you will get an error because annotations don’t store values:

>>> Point.x
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "playground.py", line 8, in __get__
    return instance.__dict__[self.name]
AttributeError: 'NoneType' object has no attribute '__dict__'

Great! We figured out the nature of annotations. Now let’s continue with our descriptor and decorator. In fact, descriptor TypeChecker didn’t change from the previous section, so let’s switch to decorator type_check.

The decorator changed, but just a bit. We iterate through annotations and for each of them we create a descriptor attribute. One thing is a bit tricky — we create a type checker class with proper type for each attribute.

Point behavior still didn’t change:

>>> p = Point(1, 2)
>>> print(p)
Point: {'_x': 1, '_y': 2}
>>> p.move_by(1, 2)
>>> print(p)
Point: {'_x': 2, '_y': 4}
>>> p = Point('1', '2')
Traceback (most recent call last):
  File "playground.py", line 47, in <module>
    p = Point('1', '2')
  File "playground.py", line 31, in __init__
    self.x = x
  File "playground.py", line 12, in __set__
    'Booooo! Expecting a {}'.format(self.required_type.__name__)
AssertionError: Booooo! Expecting a int

We created an elegant solution by using annotations, decorators and no descriptors anymore. We could stop right here, but let’s check what we can do with metaclasses.

Metaclasses

If you are not familiar with metaclasses, you can check out this article. In short, a metaclass is defined as “the class of a class”. Any class whose instances are themselves classes, is a metaclass. This way metaclasses are used to construct classes (creation and initialisation). To control creation you can implement metaclass’s method __new__, and to control initialisation you can implement __init__ constructor.

Using metaclasses is tricky and should be avoided in the majority of cases. The following code is more complicated and definitely is not the best option for type checks but let’s try it anyway:

class TypeChecker:
    required_type = object
 
    def __init__(self, name=None):
        self.name = '_{}'.format(name)
 
    def __get__(self, instance, owner=None):
        return instance.__dict__[self.name]
 
    def __set__(self, instance, value):
        assert isinstance(value, self.required_type), \
               'Booooo! Expecting a {}'.format(self.required_type.__name__)
        instance.__dict__[self.name] = value
 
 
def type_check(cls):
    for var_name, var_type in cls.__annotations__.items():
        class Checker(TypeChecker):
            required_type = var_type
 
        setattr(cls, var_name, Checker(var_name))
    return cls
 
 
class TypeCheckMeta(type):
    def __new__(meta, name, bases, dct):
        cls = super().__new__(meta, name, bases, dct)
        return type_check(cls)
 
 
class Point(metaclass=TypeCheckMeta):
    x: int
    y: int
 
    def __init__(self, x, y):
        self.x = x
        self.y = y
 
    def move_by(self, dx, dy):
        self.x += dx
        self.y += dy
 
    def __str__(self):
        return 'Point: {}'.format(self.__dict__)

As you can see, TypeChecker and type_check stay untouched. We just added TypeCheckMeta and use it as metaclass of Point.

The TypeCheckMeta is responsible for Point creation. In our case metaclass modifies our point class inside __new__ method by applying the type_check decorator.

The same as in previous sections, point works as expected:

>>> p = Point(1, 2)
>>> print(p)
Point: {'_x': 1, '_y': 2}
>>> p.move_by(1, 2)
>>> print(p)
Point: {'_x': 2, '_y': 4}
>>> p = Point('1', '2')
Traceback (most recent call last):
  File "playground.py", line 52, in <module>
    p = Point('1', '2')
  File "playground.py", line 36, in __init__
    self.x = x
  File "playground.py", line 12, in __set__
    'Booooo! Expecting a {}'.format(self.required_type.__name__)
AssertionError: Booooo! Expecting a int

As I mentioned before, metaclasses are tricky and it’s better to avoid using them. They can do implicit changes which can become a surprise to other developers, e.g. in case of inheritance.

Conclusion

We considered multiple type checking solutions. Personally, I prefer the one with annotations. As a developer, I just have to add annotations and use a decorator: it’s quite easy. My code stays clean and readable!