Some Tips and Tricks of Python (Pt. II)

[Photo on pngfly.com]

Functions in Python can also be full of various ‘tricks’.
Let me show you what I mean below:

Note: Examples are for python3 only, for python2 there are some differences in method names, but the concept remains the same.

Object methods magic

In order to create a method in Python class you can simply use:

class A:
    def m(self):
        print('hello world!')
a = A()
a.m()
hello world

Now, let’s examine what exactly happens here. Despite the apparent simplicity, this method is stocked with several interesting things.

a.m
<bound method A.m of <__main__.A object at 0x105d7b0f0>>

Compare with a function:

def f():
    pass
f
<function __main__.f()>

bound method is not a just title. It means that a.m isn’t m function, but a function with context a, which has a reference to a via self inside. Let’s rewrite A class:

class A:
    def m(self):
        print(self)
a = A()
a.m()
<__main__.A object at 0x105cd7358>
a
<__main__.A at 0x105cd7358>

Value 0x105cd7358 (object id) means that in both cases we deal with the same object a. Method m will be related with context a even if only to assign it to some variable:

f = a.m
f()
<__main__.A object at 0x105cd7358>

Bound method m is a complex object, which knows about function m on one side, and about context (object) a on the other:

a.m.__func__
<function __main__.A.m(self)>
a.m.__self__
<__main__.A at 0x105cd7358>

If one tries to call a.m.__func__ it will throw an error:

a.m.__func__()
TypeError: m() missing 1 required positional argument: 'self'

When you call a.m it will inject a as self and under the hood it’s equal to:

a.m.__func__(a.m.__self__)
<__main__.A object at 0x105cd7358>

So, the bound method just hides that within its call it then calls the original function inside itself and injects its context (object) as the first argument.

The self in method can be renamed to whatever you wish as it’s just a python convention:

class A:
    def m(me):
        print(me)
a = A()
a.m()
<__main__.A object at 0x105d5c438>

Getting access to m function is possible via A as well:

A.m
<function __main__.A.m(me)>
a.m.__func__
<function __main__.A.m(me)>

And as we see A.m is not a bound method (no related context, because no created object at all), but just a function under namespace class A:

A.m(1)
1
A.m('hello')
hello
A.m(a)
<__main__.A object at 0x105d5c438>

So the code, which is executed under namespace class A, is the regular python code:

class A:
    def m(self):
        pass
    n = m
a = A()
a.m
<bound method A.m of <__main__.A object at 0x105cd74e0>>
a.n
<bound method A.m of <__main__.A object at 0x105cd74e0>>

We can use assignments, conditions, cycles, and deletions under class declaration and it will work. For example:

class A:
    def m(self):
        pass
    n = m 
    del m
a = A()
a.n
<bound method A.m of <__main__.A object at 0x105d742b0>>
a.m
AttributeError: 'A' object has no attribute 'm'

By default, python also allows the option to assign methods to class outside of class declaration:

class A:
    pass
def m(self):
    print(self)
A.m = m
a = A()
a.m()
<__main__.A object at 0x105d689e8>

It is not possible to create an object method with a simple assignment, because object method is a bound method, as shown here:

class A:
    def m(self):
        print(self)
a = A()
def n(self):
    print(self)
a.n = n
a.n()
TypeError: n() missing 1 required positional argument: 'self'
a.n
<function __main__.n(self)>
a.m
<bound method A.m of <__main__.A object at 0x105d76198>>

In the example above, n is not a contextually bound method, but rather a function under namespace a.

But, it is possible to create an instance method with assignment via some tricks like:

import types
a.n = types.MethodType(n, a)
a.n
<bound method n of <__main__.A object at 0x105d76198>>

This method isn’t very popular and is rarely used — usually only for some tricks or patches.

Another interesting aspect is that reference a.m.__func__ is a read-only:

class A:
    def m(self):
        print('hello')
a = A()
def m(self):
    print('bye-bye')
a.m.__func__ = m
AttributeError: readonly attribute

So, by default, it’s prohibited to change reference of bound method a.m to original function m, but of course it’s possible to change A.m:

A.m = m
a.m()
bye-bye

Or to change function byte-code:

class A:
    def m(self):
        print('hello')
a = A()
def m(self):
    print('bye-bye')
a.m.__func__.__code__ = m.__code__
a.m()
bye-bye

Pay attention, as when we change original function under a class, it will affect all instances of the class!

Tricks with Decorators

I’m sure you’ve seen a similar code in python:

@decor_func
def some_func(some_args):
    # some code here

It’s called ‘decorating’ and decor_func is a corresponding ‘decorator’. Decorators are also used to make a wrapper over original (decorated) functions and to add some extra actions before or after it. For example:

def my_decor(func):
    def wrapper(*args, **kwgs):
        print("before")
        result = func(*args, **kwgs)
        print("after")
        return result
    return wrapper
@my_decor
def func():
    print("hello world")
func()
before
hello world
after

A notation with @ is a syntax sugar which originally means:

func = my_decor(func)

So the decorating formula is:

a = f(a)

where both a & f are callable objects. It means that decorator and decorated object can be represented as a function or a class.

In most cases it implies that the decorator returns wrapper over original function or class, but it can actually return anything:

def d(func):
    return "hello world"
@d
def f():
    return 1
f()
TypeError: 'str' object is not callable
f
'hello world'

In most cases it will be useless so the common usage pattern is to return a callable wrapper inside decorator, in order to use the decorating callable object further.

As a decorator returns a wrapper in most situations, this wrapper hides details of the original function like its documentation, for example:

def d(func):
    def w(*args, **kwgs):
        return func(*args, **kwgs)
    return w
@d
def f():
    """Prints hello world."""
    print('hello world')
f
<function __main__.d.<locals>.w(*args, **kwgs)>
f.__name__
'w'
help(f)
w(*args, **kwgs)

As you can see, this is not transparent decorating and adds confusing details to f.

In order to avoid it for proper decorating, it is recommended to use the additional function:

from functools import wraps
def d(func):
    @wraps(func)
    def w(*args, **kwgs):
        return func(*args, **kwgs)
    return w
@d
def f():
    """Prints hello world."""
    print('hello world')
f
<function __main__.f()>
f.__name__
'f'
help(f)
f() Prints hello world.

With this, everything should now be ok.

We also can decorate a class in order to add extra methods, for example:

def d(cls):
    def m(self):
        print(self)
    cls.m = m
    return cls

@d
class C:
    """My class."""
c = C()
c.m()
<__main__.C object at 0x105e244e0>
help(c)
class C(builtins.object)
|  My class.
|
|  Methods defined here:
|
|  m(self)

wraps don’t need a class case because we return the same class, not a wrapper.

Make a class decorator is a bit trickier than a function, since class call returns the instance:

class D:
    def __init__(self, func):
        self.func = func
@D
def f():
    print('hello world')
f()
TypeError: 'D' object is not callable
f
<__main__.D at 0x105ef19b0>

In order to make f callable we should add a special method to D to make its instances callable:

class D:
    def __init__(self, func):
        self.func = func
    def __call__(self, *args, **kwgs):
        print(self)
        return self.func(*args, **kwgs)
@D
def f():
    print('hello world')
f()
<__main__.D object at 0x105ef19e8>
hello world
f
<__main__.D at 0x105ef19e8>

wraps can’t be used here as it is used inside a function decorator.

So using a class as decorator is rare and only when it needs to get an instance of class, which can then use related context on call.

In the above example we saw usage of __call__ which can also be used to make an object a decorator:

from functools import wraps
class D:
    def __call__(self, func):
        @wraps(func)
        def w(*args, **kwgs):
            print(self)
            return func(*args, **kwgs)
        return w
d = D()
@d
def f():
    print('hello world')
f()
<__main__.D object at 0x105cbfda0>
hello world

Or just:

@D()
def f():
    print('hello world')
f()
<__main__.D object at 0x105d63ef0>
hello world

As we see, we can add some function or class in decorating, but after the call it should return callable object!

Such approach of decorating is called decorators factory, which creates a new unique decorator on each decorating case. For example:

from functools import wraps
from random import random
def uniq():
    seed = random()
    def d(func):
        @wraps(func)
        def w(*args, **kwgs):
            print('seed', seed)
            return func(*args, **kwgs)
        return w
    return d
@uniq()
def f():
    print('hello')
@uniq()
def g():
    print('bye-bye')
f()
seed 0.41581692246199853
hello
g()
seed 0.5093708771705259
bye-bye
f()
seed 0.41581692246199853
hello
g()
seed 0.5093708771705259
bye-bye

Of course, any callable object decorators factory can pass arguments if you need.

Class methods can be decorated too (as we see above, they are just functions under namespace). For example:

from functools import wraps
def d(func):
    @wraps(func)
    def w(self, *args, **kwgs):
        self.m()
        return func(self, *args, **kwgs)
    return w
class C:
    def m(self):
        print('hello world')
    @d
    def n(self):
        print('bye-bye')

c = C()
c.n()
hello world
bye-bye

Interestingly, func refers to the original n function and not to the bound method. This happens because decorator d is called inside class definition, when no bound methods exist, but there are only functions under namespace class C. That’s why we need to pass context self to func explicitly:

from functools import wraps
def d(func):
    @wraps(func)
    def w(self, *args, **kwgs):
        self.m()
        print(func)                      # Added print here!
        return func(self, *args, **kwgs)
    return w
class C:
    def m(self):
        print('hello world')
    @d
    def n(self):
        print('bye-bye')
c = C()
c.n()
hello world
<function C.n at 0x105f73730>
bye-bye

Pay attention to function C.n, not the bound method C.n.

We talked above, that python code is executed under class namespace as well, so we could keep the decorator under the class:

from functools import wraps
class C:
    def d(func):
        @wraps(func)
        def w(self, *args, **kwgs):
            self.m()
            return func(self, *args, **kwgs)
        return w
    def m(self):
        print('hello world')
    @d
    def n(self):
        print('bye-bye')
c = C()
c.n()
hello world
bye-bye

But in such cases we have a side effect:

c.d
<bound method C.d of <__main__.C object at 0x105eeb5c0>>

We don’t want to have d as a bound method, all we need is a decorator and then we should hide the decorator after usage:

from functools import wraps
class C:
    def d(func):
        @wraps(func)
        def w(self, *args, **kwgs):
            self.m()
            return func(self, *args, **kwgs)
        return w
    def m(self):
        print('hello world')
    @d
    def n(self):
        print('bye-bye')
    del d
c = C()
c.n()
hello world
bye-bye
c.d
AttributeError: 'C' object has no attribute 'd'

All these extra actions give code a bad look. This is why method decorators are defined outside of class — to make code clean and readable.

Getter / setter magic

Methods for decorating are commonly used patterns in python in order to get dynamically calculated properties of an object. For this there is a special function property:

class C:
    @property
    def m(self):
        return 1 + 2
c = C()
c.m
3
c.m
3

As we see, @property turns a method to dynamic property, which it does not need to call.

Such a ‘getter’ is often used, but @property can also be used to add ‘setter’ or ‘deleter’:

class C:
    def __init__(self):
        self._m = None
    @property
    def m(self):
        print('get m')
        return self._m
    @m.setter
    def m(self, val):
        print('set m')
        self._m = val
    @m.deleter
    def m(self):
        print('delete m')
        del self._m
c = C()
c.m = 1
'set m'
c.m
'get m'
1
del c.m
'delete m'

After examining the above examples, you likely have two questions:

1. How has m got methods setter & getter if it originally was a function?
2. Why has it the same name def m each time?

Firstly, let’s start with the second point and rewrite class as:

class C:
    def __init__(self):
        self._m = None
    def m(self):
        print('get m')
        return self._m
    m = property(m)
    def m(self, val):
        print('set m')
        self._m = val
    m = m.setter(m)
    def m(self):
        print('delete m')
        del self._m
   m = m.deleter(m)

So m name was used in order to get a final method with name m after m = m.deleter(m) (or after m = m.setter(m) if deleter isn’t used).

And now the initial point — why the function became an object. This is because property is a class decorator and returns an instance of property with closured original m function, which is used as getter. Despite the property being built-in, we can still get its analogue with pure python:

class property(object):
    "Emulate PyProperty_Type() in Objects/descrobject.c"

    def __init__(self, fget=None, fset=None, fdel=None, doc=None):
        self.fget = fget
        self.fset = fset
        self.fdel = fdel
        if doc is None and fget is not None:
            doc = fget.__doc__
        self.__doc__ = doc

    def __get__(self, obj, objtype=None):
        if obj is None:
            return self
        if self.fget is None:
            raise AttributeError("unreadable attribute")
        return self.fget(obj)

    def __set__(self, obj, value):
        if self.fset is None:
            raise AttributeError("can't set attribute")
        self.fset(obj, value)

    def __delete__(self, obj):
        if self.fdel is None:
            raise AttributeError("can't delete attribute")
        self.fdel(obj)

    def getter(self, fget):
        return type(self)(fget, self.fset, self.fdel, self.__doc__)

    def setter(self, fset):
        return type(self)(self.fget, fset, self.fdel, self.__doc__)

    def deleter(self, fdel):
        return type(self)(self.fget, self.fset, fdel, self.__doc__)

As we can see, after decorating m = property(m) it now has the methods getter, setter and deleter.

setter and deleter aren’t as popular, possibly because of a problem related with inheritance:

class A:
    def __init__(self):
        self._x = 0
    @property
    def x(self):
        return self._x
    @x.setter
    def x(self, val):
        self._x = val
class B(A):
    @property
    def x(self):
        return self._x + 1

b = B()
b.x
1
b.x = 2
AttributeError: can't set attribute

This happens because x creates a new instance of property and for the new instance no setter is defined. It then needs to rewrite a setter even if it should just be the same. This is logical but… unfortunate.

Let’s now return back class property and make sure that it also has methods __get__, __set__, __delete__. If a class has any of these methods, it means that it implements description protocol and can be used to create dynamically calculated instance properties. For example:

class Prop:
    def __get__(self, obj, objtype):
        print('object is', obj)
        print('object type is', objtype)
    def __set__(self, obj, value):
        print('object is', obj)
        print('assigned value is', value)

class A:
    m = Prop()
a = A()
a.m
object is <__main__.A object at 0x105ef12b0>
object type is <class '__main__.A'>
a.m = 1
object is <__main__.A object at 0x105ef12b0>
assigned value is 1

Using descriptors, it’s possible to implement native decorators @classmethod and @statismethod with pure python:

import types
class my_classmethod:
    def __init__(self, func):
        self.__func__ = func
    def __get__(self, obj, objtype=None):
        return types.MethodType(self.__func__, objtype or type(obj))
class my_staticmethod:
    def __init__(self, func):
        self.__func__ = func
    def __get__(self, obj, objtype=None):
        return self.__func__
class X:
     @classmethod
     def a(cls):
         pass
     @my_classmethod
     def b(cls):
         pass
     @staticmethod
     def c():
         pass
     @my_staticmethod
     def d():
         pass
x = X()
x.a
<bound method X.a of <class '__main__.X'>>
x.b
<bound method X.b of <class '__main__.X'>>
x.c
<function __main__.X.c()>
x.d
<function __main__.X.d()>

As you see custom implementation behaviour corresponds to native.

To be continued…

Many kudos for text review & comments to David Lorbiecke