Photo by Aziz Acharki on Unsplash

Python Magic Methods Explained

Everything we need to know about magic methods, which are not so magical.

Magic Methods — Not so magical!! 🃏

In Python, method names that have leading and trailing double underscores are reserved for special use like the__init__ method for object constructors, or __call__ method to make object callable. These methods are known as dunder methods. dunder here means “Double Under (Underscores)”. These dunder methods are often referred to as magic methods — Although there is nothing magical about them. Many in Python community don’t like that word (magic), since it gives a feeling that the use of these methods is discouraged, however quite the opposite is true.

Why hassle of wrapping them in double underscores?

The choice of wrapping these functions with double-underscores on either side was really just a way of keeping the language simple. The Python creators didn’t want to steal perfectly good method names from us(such as call or iter), but they also did not want to introduce some new syntax just to declare certain methods “special”. The dunders achieve the desired objective of making certain methods special while also making them just the same as other plain methods in every aspect except naming convention.

Things to Remember for dunders

1. Call them “dunders” — Since there is nothing arcane or magical about them. Terminology like “magic” makes them seem much more complicated than they actually are.
2. Implement dunders as desired — It’s a core Python feature and should be used as needed.
3. Inventing our own dunders is highly discouraged — It’s best to stay away from using names that start and end with double underscores in our programs to avoid collisions with our own methods and attributes.

Dunder methods can be used to emulate behaviour of built-in types to user defined objects. Consider the following example where we add len() method support to our own object.

class NoLenDefined:
    pass
>>> obj = NoLenDefined()
>>> len(obj)
TypeError: "object of type 'NoLenDefined' has no len()"

Adding the __len__() dunder method will fix the error.

class LenDefined:
    def __len__():
        return 1
>>> obj = LenDefined()
>>> len(obj)
1

🖊 NOTE: len() internally calls the special method __len__() to return the length of the object.
💡 Tip: You can use the dir() method on the object to see the dunder methods inherited by the class. Example: dir(int)

Let’s look at various “dunder” methods to have the better understanding of various features Python provides.

Object Initialization: __init__

When an object is created, it is initialized by calling the __init__ method on the object.

class Person:
    def __init__(self, name, age):
        self.name = name
        self.age = age
>>> person = Person('Sarah', 25)
>>> person
<__main__.Person instance at 0x10d580638>

When the __init__ method is invoked, the object (in this case, person) is passed as “self”. Other arguments used in the method call are passed as the rest of the arguments to the function.

Object Representation: __str__ , __repr__

When we define a custom class and try to print its instance to the console, the result does not describe the object well, since the default “to string” conversion is basic and lacks details. Let’s consider the following example:

class Person:
    def __init__(self, name, age):
        self.name = name
        self.age = age
>>> person = Person('Sarah', 25)
>>> print(person)
<__main__.Person instance at 0x10d580638>
>>> person
<__main__.Person instance at 0x10d580638>

By default, we got the name of the class along with the id of the object. It would be more desirable to get the attributes of the object printed such as this:

print (person.name, person.age)
Sarah 25

To accomplish this, we can add our own to_string() method but doing that would be overlooking the python in built mechanism of representing objects as strings. Therefore let’s add “dunder” methods to our class to describe our object as we want.

class Person:
    def __init__(self, name, age):
        self.name = name
        self.age = age
    def __str__(self):
        return "Person: {}, Age: {}".format(self.name, self.age)
>>> person = Person('Sarah', 25)
>>> print(person)
Person: Sarah, Age: 25
>>> person
<__main__.Person instance at 0x10d5807e8>

Therefore, __str__ method can be overridden to return a printable string representation of any user defined class.

>>> print(person)
Person: Sarah, Age: 25
>>> str(person)
Person: Sarah, Age: 25
>>> '{}'.format(person)
Person: Sarah, Age: 25

__repr__ is similar to __str__ but is used in a different situation. If we inspect our person object in interpreter session, we still got the <__main__.Person instance at 0x10d5807e8> output. Let’s redefine our class to contain both dunder methods.

class Person:
    def __init__(self, name, age):
        self.name = name
        self.age = age
    def __str__(self):
        print('inside str')
        return "Person: {}, Age: {}".format(self.name, self.age)
    def __repr__(self):
        print('inside repr')
        return " Person: {}, Age: {}".format(self.name, self.age)
>>> person = Person('Sarah', 25)
>>> person
inside repr
Person: Sarah, Age: 25
>>>print(person)
inside str
Person: Sarah, Age: 25

As seen above, __repr__ is invoked, when object is inspected in interpreter session.
On a high level, __str__ is used for creating output for end user while __repr__ is mainly used for debugging and development. repr’s goal is to be unambiguous and str’s is to be readable.

Highlights for __str__ , __repr__

1. We can control to-string conversion in our own classes using the __str__ and __repr__ “dunder” methods
2. __repr__ compute the “official” string representation of an object (having all information about the object) and __str__ is used for “informal” string representation of an object.
3. If we don’t add a __str__ method, Python falls back on the result of the __repr__ when searching for __str__ . Therefore adding __repr__ to our classes is recommended.

Iteration: __getitem__ , __setitem__ , __len__

Python built-in types list, str, and bytes can use the slicing operator [] to access range of elements. Implementing __getitem__, __setitem__in a class allows its instances to use the [] (indexer) operator. Therefore, the __getitem__ and __setitem__dunder methods is used for list indexing, dictionary lookups, or accessing ranges of values. To grasp the concept better, let’s consider the example in which we create our own custom list.

import random as ran
class CustomList:
  def __init__(self, num):
    self.my_list = [ran.randrange(1,101,1) for _ in range(num)]
>>> obj = CustomList(5)
>>> obj.my_list
[59, 83, 96, 86, 59]
>>> len(obj)
AttributeError:
>>> for no in obj:
...    print (no)
AttributeError:
>>> obj[1]
AttributeError:

With the above class definition, we cannot iterate over our object since the above statements raises an AttributeError . Let’s implement the dunder methods in our class to make it iterable.

import random as ran
class CustomList:
  def __init__(self, num):
    self.my_list = [ran.randrange(1,101,1) for _ in range(num)]
  
  def __str__(self):
    return str(self.my_list)
  def __setitem__(self, index, value):
    self.my_list[index] = value
  def __getitem__(self, index):
    return self.my_list[index]
  def __len__(self):
    return len(self.my_list)
>>> obj = CustomList(5)
>>> print(obj)
[59, 83, 96, 86, 59]
>>> len(obj)
5
>>> obj[1]
83
>>> for item in obj:
...    print (item)
59
83
96
86
59

Therefore using __setitem__ , __getitem__ , __len__ dunder methods allows us to use slicing operator and makes our object iterable.
NOTE: __iter__ and __next__ dunder methods are used to write iterable objects as well but they are out of scope for discussion here and we will be discussing them in a separate post. 😄

Object Invocation: __call__

We can make any object callable like a regular function by adding the __call__ dunder method. Let’s consider the below toy(not very meaningful) example just to showcase the __call__ method.

class Person:
    def __init__(self, name, age):
        self.name = name
        self.age = age
    def __call__(self):
        print ('Person: {}, Age: {}'.format(self.name, self.age))
>>> person = Person()
>>> person()
Person: Sarah, Age: 25

__call__ can be particularly useful in classes with instances that needs to often change state. "Calling" the instance can be an intuitive and elegant way to change the object's state. An example might be a class representing an entity's position on a plane:

class Entity:
    '''Callable to update the entity's position.'''

    def __init__(self, size, x, y):
        self.x, self.y = x, y
        self.size = size

    def __call__(self, x, y):
        '''Change the position of the entity.'''
        self.x, self.y = x, y
>>> point = Entity(10, 20)
>>> print(point.x, point.y)
10 20
>>> point(30, 40)
>>> print (point.x, point.y)
30 40

Generally __call__ is used whenever we want to provide a simple interface that resembles a simple function. However, it can be hard to see through the same.

Conclusions

Dunder methods can be used to emulate behaviour of built-in types to user defined objects and is core Python feature that should be used as needed. 🙌
We have touched upon frequently used dunder methods. These methods help in writing feature-rich, elegant, and easy-to-use classes.There are plethora of dunder methods available in Python language. To read more about them, Python reference documentation is the best place to dig.
On a closing note, the absurd degree of control, dunder method provides, makes us wonder sometimes that they are indeed magical 😉