Demystifying the Black Magic in Python: Metaclasses and Class Creation
“Metaclasses are deeper magic than 99% of users should ever worry about. If you wonder whether you need them, you don’t (the people who actually need them know with certainty that they need them, and don’t need an explanation about why).”
— Tim Peters
The striking quote above was once said by Tim Peters, one of the most influential python developers ever, the inventor of the TimSort (the sorting algorithm used in python) as well as the author of the renowned The Zen of Python [1]. If you are like me though, you would be more motivated than before to learn more about metaclasses after reading this quote. Who knows, maybe you could be the next person to be a member of the privileged 1% minority, precisely knowing when and how to use the metaclasses. A metaclass is actually a class whose instances are classes themselves. Thanks to them, you can manipulate class creation stages to get behavior that is not possible otherwise. In fact, a rather realistic reason to learn more about the metaclasses is if you need to write or understand a framework that needs to provide some complicated service in a clean interface. In many cases, the high-level features of the language known by the most newbie or less experienced users are not enough. These features are simply not powerful enough to handle the task at hand, or result in a somewhat messy interface which is not a desirable property to have in a framework at all. Luckily, we have some mysterious mechanisms like metaclasses that are built into the language itself that can be used to help us in such cases. Indeed, as we will read from the detailed explanation below, the metaclasses are really at the core of the language and essential to the python’s class mechanism. In fact, this is the other major reason to learn more about the metaclasses since they are essential to really understand python’s class system and object creation. As commonly known, everything is an object in python, and this article will also try to explain how and why this is the case. Since objects, classes and metaclasses are all related, how the objects and classes in python are created can not be truly understood without understanding metaclasses, I will try to explain all these concepts together.
Class and Object Creation
A class is a blueprint for an object in Python. When you define a class, you specify the methods and attributes (instance and class variables) bound to the class. The peculiarity of the instance variables is that they are bound to the objects inside the special __init__ method and not accessible by the class itself. However, an object instantiated from that class can access all these attributes and methods. After construction, when you try to access a variable using the object name the with dot operator e.g. obj.attr, the variable attr is first looked up at the variables bound to the object (by querying the obj.__dict__ dictionary with attr as the key). If it is not found there, it is looked up in the obj.__class__.__dict__ instead (indeed it will be there if attr is a class variable). In fact, this is the reason why an object can have instance and class variables with the same name but potentially with different values.
When you instantiate an object using a class, the first thing happening is the __call__ method of the class is invoked. In fact, this holds for any callable object, not just for classes. In other words, if you call an object X with X(…) syntax, the python interpreter will replace that statement with the __call__ method of X. The following two ways are equivalent ways of creating an empty list:
When you are creating a new object from a class (object instantiation) with the __call__ method, the __new__ method of the class is invoked on the class itself (as a class method whose first parameter is cls to signify that) to create and return a new instance. The default implementation of the __call__ ensures calling __new__. Finally, __init__ is called on the constructed object to initialize its instance attributes (and methods). In other words, __new__ allocates the memory for the object, creates it and calls __init__ to initialize its attributes. Typically, we do not override __new__, instead we use the __init__ method to control the attributes of the created object.
Metaclass — class pair has a very similar relationship to the one the class — object pair has. A metaclass is a blueprint for the class, we create a class object using a metaclass. (Remember everything in python is an object, all of the classes or metaclasses as well). We control the class creation using the __new__ and __init__ (mostly preferred) methods of the metaclass. First, __new__ is called on the metaclass to create the class object. Therefore, its first argument is mcls by convention to signify on which object it is invoked. Inside the __new__ method, __init__ is called on the constructed class object. Since it is invoked on the class object, the first argument of the __init__ is cls. Below diagram summarizes the relationship among the regular object (instance), class and metaclass.
It is very likely that you have been creating your classes using the familiar syntax of starting with the class keyword, followed by the class name and then listing the methods etc below. In fact this is informally called static class creation or rather ‘creating classes statically’ to separate this process from the static class[2] concept found in some languages like Java, which is a completely different thing. In fact, python offers another alternative which is known as creating classes dynamically. This alternative way requires playing with the metaclasses as we will see shortly. In fact, the metaclasses are at the scene during the class creation through both alternative ways but the static definition of classes (through class keyword) mostly hides it. The reason is that the default metaclass is implicitly used and hence is not required to be specified. However, for the dynamic creation, we need to have a callable object to create the class, hence the metaclass is explicitly written even if the default one is used. This default metaclass is type. Moreover, it is the base class for all other metaclasses, so all metaclasses inherit from the type itself or one of its subclasses.
Now, let’s explore the two ways of class creation with examples. Assuming C is its name, B1 and B2 are its parent classes from which it inherits, and M is its metaclass, the following two are equivalent ways of constructing the class C.
1- Using the regular class C(B1, B2, metaclass=M): … syntax to create a class statically.
2- Calling M("C", (B1, B2), attr_dict) to create the class dynamically where attr_dict is the dictionary containing all attributes, i.e. fields and methods bound to the class.
Note that, if we do not specify the metaclass, type is always used as the metaclass by default in both ways of class creation. Therefore, the following three ways are equivalent if we are not using a custom metaclass:
class C(B1, B2): …
class C(B1, B2, metaclass=type): …
type("C", (B1, B2), attr_dict)
Note that the attr_dict includes all instance methods (these are provided just like regular unbound functions taking self as the first argument) as well as the instance variables. The methods include the special magic methods like __init__, which can later be used for object creation using the class we are currently constructing. The instance variables are defined inside the provided __init__ function included in the attr_dict. For example, let’s look at the definition of some custom classes using the regular syntax:
The code snipped above statically creates a class that inherits from two classes, B1 and B2. Since we did not specify a metaclass in the class definition above, type was used by default. Let’s create the class C again, this time dynamically, by calling its metaclass, type.
As you can see from the code above, the initialize_C function takes self as the first parameter, which represents the object that will later be instantiated by C. Moreover, the favorite_color is also defined as an instance method that will operate on C’s instances.
The Method Call Chain During Class Creation
When an object is called with obj.__call__(…) or obj(…) syntax, the __call__ method defined in that object’s class is invoked. This __call__ method is written inside the class definition for a regular object whereas inside the metaclass definition for a class object. Finally, if the caller object is a metaclass (e.g. when we want to create a class using that metaclass), type’s built-in __call__ implementation is invoked. For the last case, the default implementation provided by type invokes the __new__ and __init__ method of the metaclass that is responsible for creation of the class.
Since M is a metaclass, it is a callable object. Therefore, calling it with parentheses, e.g. with the M(classname, superclasses, attr_dict) statement as we did above, will invoke its __call__ method. In other words, the class creation statements we presented above are translated into M.__call__("C", (B1, B2), attr_dict). Then, the default __call__ method provided by the type (since it is the default metaclass of any metaclasses that don’t specify a custom one) invokes the __new__ and __init__ methods with the correct arguments. In other words, the following two methods are invoked from M.__call__ sequentially:
1. M.__new__(M, "C", (B1, B2), attr_dict)
Note that __new__ is a static method of M. The first parameter is always the class whose instance was requested (e.g. by calling the constructor expression with M.__call__(…) or M(…), or with usual class definition syntax). The remaining arguments are passed to the constructor expression to initialize the constructor parameters. The return value of __new__ is the new object instance, in this case a class object, C.
Note: the signature of __new__ when defined inside a metaclass is as follows:
(mcls, clsname, superclasses, attr_dict) -> object
2. M.__init__(cls, 'C', (B1, B2), attr_dict)
where cls represents the new object created by __new__ in the previous step, which is the class object C in our example.
The 1st method, __new__ creates and returns the new class C, and the __init__ initializes it (sets its all class attributes, defines object methods and attributes).
Another Use of type
Although type is the base class for all metaclasses in python (or the default metaclass for all the classes), there is another use of the type keyword in python, which acts as a regular function. This type function returns the type of its argument. In python3, class and type concepts are mostly used interchangeably. Each object has a class (or type) from which it was derived. When type is called with an instance of a class (which is a plain object), it returns the class. Instead, if it is called with a class object, it will return the name of the metaclass of that class. In other words, the type (or class) of a class is the metaclass using which it was created. The following example can explain the type function better.
We can see that the type of 5 is int, which is a class whereas the type of int is type, which is the default metaclass. By the way, instead of calling the type function, we can use the __class__ attribute to check the type of our object, which is available to all objects (including the classes or metaclasses since they are all objects as well.). To check the class of 5, we have to use parentheses though, since the dot symbol defines a float value when it is following an integer.
The above examples show that the type and class is really the same.
Python has even a built-in mechanism used to verify the type of an object. This mechanism is the isinstance function which returns True if its first argument is an instance of the second argument. Continuing our example, the following results should be expected.
This shows that 5 is an instance of int, and int is an instance of type. Note that, here type is the default metaclass, not the function presented in this section. While talking about the type metaclass, let’s check its type (class) just for fun.
Oh, that was surprising, wasn’t it? Well, actually not very much. The output above means the type of the type is the type itself. Since type is a class (actually a metaclass as it is used for instantiation of new classes), we need a metaclass to create it. It turns out that the metaclass to create the type metaclass is the type itself. We can verify that the class of type is type with the __class__ attribute as well.
Or, equivalently, the following statement verifies that type is an instance of type.
Inheritance Hierarchy vs Metaclass Hierarchy
Python’s metaclass mechanism design leads to some further interesting results. For example, let’s think of a custom metaclass which is directly derived from type. Remember, to create a metaclass we have to extend type, or any of its subclasses. The following is an example of such a custom metaclass.
Let’s check its type (or __class__ attribute):
Okay, that was expected. What about its base classes (the classes it inherits from)? There is a __bases__ attribute in python that lets us query the direct parent (also called `super`) classes of a class. To clarify, when you have a class B derived from A, and a class C derived from B, if you call __bases__ on C, it will return only B, since it is the direct parent of C. Moreover, in the case of multiple inheritance, __bases__ will return all direct parents. Going back to the parents of our example metaclass:
The above statement says that the base class of the CustomMeta is type, which is its metaclass! In other words, CustomMeta has to inherit from type (to be a metaclass) and must use type as its metaclass (to be instantiated) simultaneously. You can even go further while looking up CustomMeta’s base classes to inspect the whole class hierarchy above it using the __mro__ attribute defined for all classes (including the metaclasses of course). __mro__ is an abbreviation for Method Resolution Order and it denotes the class hierarchy traversed when a method invoked on a class (or an instance of it). For example, if you call a method of a list object, the name of the method will be searched in the __mro__ of the list class in order. Similarly, if you call a class method of any class, the method will be searched in the __mro__ of the class again. Let’s check the __mro__ of our CustomMeta class.
The output above shows that type is indeed a superclass of the CustomMeta. Note that, object, being the base class of all python classes, will show itself in the __mro__ of any class in python. This includes the type class as well. Moreover, the first item in __mro__ is the class calling __mro__ itself, which is CustomMeta in this example. Let’s check the __mro__ of the type:
This shows that type has a single superclass, the object class which means type is directly derived from the object. Remember, the first item in the MRO is the caller class itself (here, type), so we do not take that into account as a superclass. Alternatively, we can directly use the __bases__ attributes to verify that object is indeed the only superclass of type:
These examples show that MRO depends on the inheritance hierarchy. Moreover, inheritance hierarchy is unrelated to the metaclasses. type is the metaclass of itself, but is derived from the object (just like any other class). Metaclasses are in fact real classes and have properties just as other plain classes. Similarly, classes are real objects and have properties just like any other objects. However, the converse of these statements is not true. An object does not have to be a class (i.e. class object). For example, __mro__ is only defined for classes (and also metaclasses of course) so regular objects do not have a __mro__ attribute. Similarly, a class does not have to be a metaclass. In fact, the only way a class can be a metaclass is if it inherits from type, or a subclass of type. In fact, the metaclasses form an inheritance hierarchy where type sits on the top. A note for cautious readers is that object is not in the metaclass hierarchy as it is not a metaclass although type is derived from it.
Going back to the general inheritance hierarchy of the python classes, let’s check the bases and MRO of object:
Well, as expected we get an empty tuple since object is at the top of the python’s type hierarchy so it is not derived from any class. What about its MRO?
Remember, the first class in the__mro__ tuple of a class is the class itself. As object does not inherit from any class, there is no other class in its Method Resolution Order.
Let’s also call the type function on object to check its type (or the metaclass used to instantiate it) just for fun.
The above output shows that the object class is an object that is instantiated from the metaclass type. The following definitions are equivalent to this statement.
objectis an instance oftype.
2. The class of object is type (technically type is its metaclass, since object is a class itself).
Again, the inheritance relationship is unrelated to the metaclass — class relationship. Metaclass — class relationship is actually a special case of class — object relationship such that the object that is being created is a class and the class used (as a template or blueprint) for creating that object is a metaclass.
The below figure also shows the object and type relationship in a visual way.
I refer the curious readers to [3] to read further about the type — object relationship and the class creation in C level.
An object instantiated from a class is conventionally referred to by the identifier self. Therefore, the first parameter of the methods defined inside a class that are intended to operate on the generated object is named self since they receive that object as their first argument. Similarly, the methods of a metaclass that are operating its instance (the class, represented by cls) accept cls as their first parameter to represent the class object. For example, the first argument of the __init__ in a metaclass is cls by convention (although it does not have to), and the variables defined in __init__ as bound to cls actually become class variables. These are just like the regular class variables we assign while defining a class statically and they will be accessible by the new objects instantiated from that class as well. Similarly, the regular methods defined in the metaclass will be class methods bound to the generated class, and the instances of that class can call them just like other class methods defined using the @classmethod decorator.
Acknowledgement
Special thanks to Devrim Çavuşoğlu, Fatih Cagatay Akyon and Sinan Onur ALTINUÇ for their valuable feedback.
