Adventures of an iOS developer in Python

A dive into properties, descriptors, decorators, annotations and metaclasses

Introduction

Hi. I attended the EuroPython 2018 conference in Edinburgh 23–29 July 2018. I really liked Francesco Pierfederici’s workshop called Ridiculously Advanced Python and I want to share my experiences with you.

The idea of the workshop was the implementation of type checking using the following Python features:

• properties
• descriptors
• decorators
• annotations
• metaclasses

Before we begin

Be aware that some examples of this workshop are not best practices and shouldn’t be used in real projects. They are good for the demonstration of Python features.

Please install Python 3 to make sure all code examples work as expected. Personally, I prefer pyenv tool to have multiple Python versions on single machine.

Problem statement

We have the following code snippet, it represents the simplest 2D point:

`class Point:    def __init__(self, x, y):        self.x = x        self.y = y     def move_by(self, dx, dy):        self.x += dx        self.y += dy     def __str__(self):        return 'Point: {}'.format(self.__dict__)`

Initialiser (method `__init__`) adds `x` and `y` attributes to point instance and method `move_by` modifies them. It looks clean and simple.

Let’s do a bit of programming:

`>>> p = Point(1, 2)>>> print(p)Point: {'x': 1, 'y': 2}>>> p.move_by(1, 2)>>> print(p)Point: {'x': 2, 'y': 4}`

As we can see, attributes `x`and `y`changed properly. Our class `Point` works as expected. But the problem is that it’s easy to misuse it. For instance, if we use `string`s instead of `int`s, the result is completely different and undesirable:

`>>> p = Point('1', '2')>>> print(p)Point: {'x': '1', 'y': '2'}>>> p.move_by('1', '2')>>> print(p)Point: {'x': '11', 'y': '22'}`

For Python everything looks good, it just concatenates strings. But that’s not what we actually want. We definitely miss type checking. Let’s check if we can use properties to solve this problem.

Properties

If you are not familiar with properties, you can check the Python tutorial. In short, this is a way to hide specific attribute and define setter and getter interfaces to manipulate them.

Let’s improve our `Point` class. We can turn attributes `x` and `y` to properties:

`class Point:    def __init__(self, x, y):        self.x = x        self.y = y     @property    def x(self):        return self._x     @x.setter    def x(self, value):        assert isinstance(value, int), 'Booooo! Expecting an int'        self._x = value     @property    def y(self):        return self._y     @y.setter    def y(self, value):        assert isinstance(value, int), 'Booooo! Expecting an int'        self._y = value     def move_by(self, dx, dy):        self.x += dx        self.y += dy     def __str__(self):        return 'Point: {}'.format(self.__dict__)`

Let’s call `move_by` method with `int` arguments:

`>>> p = Point(1, 2)>>> print(p)Point: {'_x': 1, '_y': 2}>>> p.move_by(1, 2)>>> print(p)Point: {'_x': 2, '_y': 4}`

Everything works as expected. Output slightly changed because properties `x` and `y` manipulate with private attributes `_x` and `_y`.

Now let’s pass `string` arguments:

`>>> p = Point('1', '2')Traceback (most recent call last):  File "playground.py", line 37, in <module>    p = Point('1', '2')  File "playground.py", line 3, in __init__    self.x = x  File "playground.py", line 12, in x    assert isinstance(value, int), 'Booooo! Expecting an int'AssertionError: Booooo! Expecting an int`

Boom! Execution was interrupted. That’s exactly what we are trying to achieve — properties `x` and `y` should accept just `int` arguments.

The actual type checks are done in property setters. Such approach is well-known, it is easy to understand and maintain. But it also has a downside: code duplication. We have to do the same check for each property over and over again. That doesn’t sound good. Let’s go further — write a descriptor.

Descriptors

If you are not familiar with descriptors, please check the documentation. In short, a descriptor is an object attribute with “binding behavior”. It specifies what happens when an attribute is referenced on a model, it allows developers to manage attribute access: set, get and delete.

Let’s do some coding. We have the following descriptors:

`class TypeChecker:    required_type = object     def __init__(self, name):        self.name = '_{}'.format(name)     def __get__(self, instance, owner=None):        return instance.__dict__[self.name]     def __set__(self, instance, value):        assert isinstance(value, self.required_type), \               'Booooo! Expecting a {}'.format(self.required_type.__name__)        instance.__dict__[self.name] = value  class IntType(TypeChecker):    required_type = int`

The `TypeChecker` defines the getter and setter. The getter just returns an attribute from the instance (point in our case). The setter stores a new value as attribute in the instance and it also does the type check and interrupts execution in case the value has a wrong type.

The `TypeChecker`’s attribute `required_type` holds the actual type for type checking. By default, it’s `object`, so any type is valid. The `IntType` is a subclass of `TypeChecker`, it just defines required type. In this case it’s `int`.

Let’s modify our Point class a bit:

`class Point:    x = IntType('x')    y = IntType('y')     def __init__(self, x, y):        self.x = x        self.y = y     def move_by(self, dx, dy):        self.x += dx        self.y += dy     def __str__(self):        return 'Point: {}'.format(self.__dict__)`

As you can see, we added `x` and `y` attributes to the `Point` class itself. These attributes are instances of descriptor `IntType`. It means that Python calls getter and setter of proper descriptor when we manipulate with `x` and `y` attributes of `Point` instance.

Let’s check if everything works as expected:

`>>> p = Point(1, 2)>>> print(p)Point: {'_x': 1, '_y': 2}>>> p.move_by(1, 2)>>> print(p)Point: {'_x': 2, '_y': 4}`

Yep, nothing changed. And what about `string` usage?

`>>> p = Point('1', '2')Traceback (most recent call last):  File "playground.py", line 41, in <module>    p = Point('1', '2')  File "playground.py", line 25, in __init__    self.x = x  File "playground.py", line 12, in __set__    'Booooo! Expecting a {}'.format(self.required_type.__name__)AssertionError: Booooo! Expecting a int`

Also no changes. Descriptor implements type checking and interrupts execution in case of non-`int` argument.

It looks like extra work to create own descriptors, but it will save coding efforts in the future: you don’t have to copy-paste type checking logic — you can just reuse type checker descriptors. Code readability and testability also increase.

But one thing is annoying — you have to repeat attribute name twice. E.g. `x` in `x = IntType(‘x’)`.

Let’s check if we can improve our solution by using decorators.

Decorators

If you are not familiar with decorators, you can check Python tutorial. In short, decorator allows to add any functionality to any object. In most cases that object is a function but in our case it’s a class.

Let’s extend code from previous section — add decorator `type_check` and decorate our `Point` class:

`class TypeChecker:    required_type = object     def __init__(self, name=None):        self.name = '_{}'.format(name)     def __get__(self, instance, owner=None):        return instance.__dict__[self.name]     def __set__(self, instance, value):        assert isinstance(value, self.required_type), \               'Booooo! Expecting a {}'.format(self.required_type.__name__)        instance.__dict__[self.name] = value  class IntType(TypeChecker):    required_type = int  def type_check(cls):    for var_name, checker in cls.__dict__.items():        if isinstance(checker, TypeChecker):            checker.name = '_{}'.format(var_name)    return cls  @type_checkclass Point:    x = IntType()    y = IntType()     def __init__(self, x, y):        self.x = x        self.y = y     def move_by(self, dx, dy):        self.x += dx        self.y += dy     def __str__(self):        return 'Point: {}'.format(self.__dict__)`

Definition of attributes `x` and `y` is slightly changed — the name argument is omitted. Why so? The answer lays inside decorator `type_check`. It receives our `Point` class object as an argument, iterates through the type checker attributes and sets up their name attribute.

Let’s see if we can still create our point with int arguments:

`>>> p = Point(1, 2)>>> print(p)Point: {'_x': 1, '_y': 2}>>> p.move_by(1, 2)>>> print(p)Point: {'_x': 2, '_y': 4}`

Works as it used to work. And `string` arguments?

`>>> p = Point('1', '2')Traceback (most recent call last):  File "playground.py", line 49, in <module>    p = Point('1', '2')  File "playground.py", line 33, in __init__    self.x = x  File "playground.py", line 12, in __set__    'Booooo! Expecting a {}'.format(self.required_type.__name__)AssertionError: Booooo! Expecting a int`

Nice, interruption occurred. It means that type checking is still there.

So, we simplified the definition of the `x` and `y` attributes. Our code looks a bit cleaner. And this logic is reusable, we just have to reuse our `type_check` decorator. But can we go further? Yes, we can. We just need to use annotations.

Annotations

If you are not familiar with annotations, you can check PEP 526. In short, annotations are specially-formatted comments that are associated with variables and specific parts of functions. Annotations allow to increase code readability but have no impact to execution at run time. Annotations could be used for documentation generation and code analysis.

We can reduce our code by using annotations:

`class TypeChecker:    required_type = object     def __init__(self, name=None):        self.name = '_{}'.format(name)     def __get__(self, instance, owner=None):        return instance.__dict__[self.name]     def __set__(self, instance, value):        assert isinstance(value, self.required_type), \               'Booooo! Expecting a {}'.format(self.required_type.__name__)        instance.__dict__[self.name] = value  def type_check(cls):    for var_name, var_type in cls.__annotations__.items():        class Checker(TypeChecker):            required_type = var_type         setattr(cls, var_name, Checker(var_name))    return cls  @type_checkclass Point:    x: int    y: int     def __init__(self, x, y):        self.x = x        self.y = y     def move_by(self, dx, dy):        self.x += dx        self.y += dy     def __str__(self):        return 'Point: {}'.format(self.__dict__)`

As you maybe already noticed, we have removed `IntType`, modified the `type_check` decorator a bit and turned the `x` and `y` attributes to annotations.

Before we continue, it’s worth to mention that we can get a list of class annotations by calling `__annotations__`:

`>>> print(Point.__annotations__){'x': <class 'int'>, 'y': <class 'int'>}`

But what happens if we call `__dict__`? Let’s figure it out:

`>>> print(Point.__dict__){'__module__': '__main__', '__annotations__': {'x': <class 'int'>, 'y': <class 'int'>}...`

As you can see, all annotations are accessible via separate key `__annotations__`. If you try to access annotation `x` directly as an attribute, you will get an error because annotations don’t store values:

`>>> Point.xTraceback (most recent call last):  File "<stdin>", line 1, in <module>  File "playground.py", line 8, in __get__    return instance.__dict__[self.name]AttributeError: 'NoneType' object has no attribute '__dict__'`

Great! We figured out the nature of annotations. Now let’s continue with our descriptor and decorator. In fact, descriptor `TypeChecker` didn’t change from the previous section, so let’s switch to decorator `type_check`.

The decorator changed, but just a bit. We iterate through annotations and for each of them we create a descriptor attribute. One thing is a bit tricky — we create a type checker class with proper type for each attribute.

Point behavior still didn’t change:

`>>> p = Point(1, 2)>>> print(p)Point: {'_x': 1, '_y': 2}>>> p.move_by(1, 2)>>> print(p)Point: {'_x': 2, '_y': 4}>>> p = Point('1', '2')Traceback (most recent call last):  File "playground.py", line 47, in <module>    p = Point('1', '2')  File "playground.py", line 31, in __init__    self.x = x  File "playground.py", line 12, in __set__    'Booooo! Expecting a {}'.format(self.required_type.__name__)AssertionError: Booooo! Expecting a int`

We created an elegant solution by using annotations, decorators and no descriptors anymore. We could stop right here, but let’s check what we can do with metaclasses.

Metaclasses

If you are not familiar with metaclasses, you can check out this article. In short, a metaclass is defined as “the class of a class”. Any class whose instances are themselves classes, is a metaclass. This way metaclasses are used to construct classes (creation and initialisation). To control creation you can implement metaclass’s method `__new__`, and to control initialisation you can implement `__init__` constructor.

Using metaclasses is tricky and should be avoided in the majority of cases. The following code is more complicated and definitely is not the best option for type checks but let’s try it anyway:

`class TypeChecker:    required_type = object     def __init__(self, name=None):        self.name = '_{}'.format(name)     def __get__(self, instance, owner=None):        return instance.__dict__[self.name]     def __set__(self, instance, value):        assert isinstance(value, self.required_type), \               'Booooo! Expecting a {}'.format(self.required_type.__name__)        instance.__dict__[self.name] = value  def type_check(cls):    for var_name, var_type in cls.__annotations__.items():        class Checker(TypeChecker):            required_type = var_type         setattr(cls, var_name, Checker(var_name))    return cls  class TypeCheckMeta(type):    def __new__(meta, name, bases, dct):        cls = super().__new__(meta, name, bases, dct)        return type_check(cls)  class Point(metaclass=TypeCheckMeta):    x: int    y: int     def __init__(self, x, y):        self.x = x        self.y = y     def move_by(self, dx, dy):        self.x += dx        self.y += dy     def __str__(self):        return 'Point: {}'.format(self.__dict__)`

As you can see, `TypeChecker` and `type_check` stay untouched. We just added `TypeCheckMeta` and use it as metaclass of `Point`.

The `TypeCheckMeta` is responsible for `Point` creation. In our case metaclass modifies our point class inside `__new__` method by applying the `type_check` decorator.

The same as in previous sections, point works as expected:

`>>> p = Point(1, 2)>>> print(p)Point: {'_x': 1, '_y': 2}>>> p.move_by(1, 2)>>> print(p)Point: {'_x': 2, '_y': 4}>>> p = Point('1', '2')Traceback (most recent call last):  File "playground.py", line 52, in <module>    p = Point('1', '2')  File "playground.py", line 36, in __init__    self.x = x  File "playground.py", line 12, in __set__    'Booooo! Expecting a {}'.format(self.required_type.__name__)AssertionError: Booooo! Expecting a int`

As I mentioned before, metaclasses are tricky and it’s better to avoid using them. They can do implicit changes which can become a surprise to other developers, e.g. in case of inheritance.

Conclusion

We considered multiple type checking solutions. Personally, I prefer the one with annotations. As a developer, I just have to add annotations and use a decorator: it’s quite easy. My code stays clean and readable!