Demystifying the Black Magic in Python: Metaclasses and Class Creation
“Metaclasses are deeper magic than 99% of users should ever worry about. If you wonder whether you need them, you don’t (the people who actually need them know with certainty that they need them, and don’t need an explanation about why).”
— Tim Peters
The striking quote above was once said by Tim Peters, one of the most influential python developers ever, the inventor of the TimSort (the sorting algorithm used in python) as well as the author of the renowned The Zen of Python [1]. If you are like me though, you would be more motivated than before to learn more about metaclasses after reading this quote. Who knows, maybe you could be the next person to be a member of the privileged 1% minority, precisely knowing when and how to use the metaclasses. A metaclass is actually a class whose instances are classes themselves. Thanks to them, you can manipulate class creation stages to get behavior that is not possible otherwise. In fact, a rather realistic reason to learn more about the metaclasses is if you need to write or understand a framework that needs to provide some complicated service in a clean interface. In many cases, the high-level features of the language known by the most newbie or less experienced users are not enough. These features are simply not powerful enough to handle the task at hand, or result in a somewhat messy interface which is not a desirable property to have in a framework at all. Luckily, we have some mysterious mechanisms like metaclasses that are built into the language itself that can be used to help us in such cases. Indeed, as we will read from the detailed explanation below, the metaclasses are really at the core of the language and essential to the python’s class mechanism. In fact, this is the other major reason to learn more about the metaclasses since they are essential to really understand python’s class system and object creation. As commonly known, everything is an object in python, and this article will also try to explain how and why this is the case. Since objects, classes and metaclasses are all related, how the objects and classes in python are created can not be truly understood without understanding metaclasses, I will try to explain all these concepts together.
Class and Object Creation
A class is a blueprint for an object in Python. When you define a class, you specify the methods and attributes (instance and class variables) bound to the class. The peculiarity of the instance variables is that they are bound to the objects inside the special __init__
method and not accessible by the class itself. However, an object instantiated from that class can access all these attributes and methods. After construction, when you try to access a variable using the object name the with dot operator e.g. obj.attr
, the variable attr
is first looked up at the variables bound to the object (by querying the obj.__dict__
dictionary with attr
as the key). If it is not found there, it is looked up in the obj.__class__.__dict__
instead (indeed it will be there if attr
is a class variable). In fact, this is the reason why an object can have instance and class variables with the same name but potentially with different values.
When you instantiate an object using a class, the first thing happening is the __call__
method of the class is invoked. In fact, this holds for any callable object, not just for classes. In other words, if you call an object X
with X(…) syntax
, the python interpreter will replace that statement with the __call__
method of X
. The following two ways are equivalent ways of creating an empty list:
When you are creating a new object from a class (object instantiation) with the __call__
method, the __new__
method of the class is invoked on the class itself (as a class method whose first parameter is cls
to signify that) to create and return a new instance. The default implementation of the __call__
ensures calling __new__
. Finally, __init__
is called on the constructed object to initialize its instance attributes (and methods). In other words, __new__
allocates the memory for the object, creates it and calls __init__
to initialize its attributes. Typically, we do not override __new__
, instead we use the __init__
method to control the attributes of the created object.
Metaclass — class pair has a very similar relationship to the one the class — object pair has. A metaclass is a blueprint for the class, we create a class object using a metaclass. (Remember everything in python is an object, all of the classes or metaclasses as well). We control the class creation using the __new__
and __init__
(mostly preferred) methods of the metaclass. First, __new__
is called on the metaclass to create the class object. Therefore, its first argument is mcls
by convention to signify on which object it is invoked. Inside the __new__
method, __init__
is called on the constructed class object. Since it is invoked on the class object, the first argument of the __init__
is cls
. Below diagram summarizes the relationship among the regular object (instance), class and metaclass.
It is very likely that you have been creating your classes using the familiar syntax of starting with the class
keyword, followed by the class name and then listing the methods etc below. In fact this is informally called static class creation or rather ‘creating classes statically’ to separate this process from the static class[2] concept found in some languages like Java, which is a completely different thing. In fact, python offers another alternative which is known as creating classes dynamically. This alternative way requires playing with the metaclasses as we will see shortly. In fact, the metaclasses are at the scene during the class creation through both alternative ways but the static definition of classes (through class
keyword) mostly hides it. The reason is that the default metaclass is implicitly used and hence is not required to be specified. However, for the dynamic creation, we need to have a callable object to create the class, hence the metaclass is explicitly written even if the default one is used. This default metaclass is type
. Moreover, it is the base class for all other metaclasses, so all metaclasses inherit from the type
itself or one of its subclasses.
Now, let’s explore the two ways of class creation with examples. Assuming C
is its name, B1
and B2
are its parent classes from which it inherits, and M
is its metaclass, the following two are equivalent ways of constructing the class C
.
1- Using the regular class C(B1, B2, metaclass=M): …
syntax to create a class statically.
2- Calling M("C", (B1, B2), attr_dict)
to create the class dynamically where attr_dict
is the dictionary containing all attributes, i.e. fields and methods bound to the class.
Note that, if we do not specify the metaclass, type
is always used as the metaclass by default in both ways of class creation. Therefore, the following three ways are equivalent if we are not using a custom metaclass:
class C(B1, B2): …
class C(B1, B2, metaclass=type): …
type("C", (B1, B2), attr_dict)
Note that the attr_dict
includes all instance methods (these are provided just like regular unbound functions taking self
as the first argument) as well as the instance variables. The methods include the special magic methods like __init__
, which can later be used for object creation using the class we are currently constructing. The instance variables are defined inside the provided __init__
function included in the attr_dict
. For example, let’s look at the definition of some custom classes using the regular syntax:
The code snipped above statically creates a class that inherits from two classes, B1
and B2
. Since we did not specify a metaclass in the class definition above, type
was used by default. Let’s create the class C
again, this time dynamically, by calling its metaclass, type
.
As you can see from the code above, the initialize_C
function takes self
as the first parameter, which represents the object that will later be instantiated by C
. Moreover, the favorite_color
is also defined as an instance method that will operate on C
’s instances.
The Method Call Chain During Class Creation
When an object is called with obj.__call__(…)
or obj(…)
syntax, the __call__
method defined in that object’s class is invoked. This __call__
method is written inside the class definition for a regular object whereas inside the metaclass definition for a class object. Finally, if the caller object is a metaclass (e.g. when we want to create a class using that metaclass), type
’s built-in __call__
implementation is invoked. For the last case, the default implementation provided by type
invokes the __new__
and __init__
method of the metaclass that is responsible for creation of the class.
Since M
is a metaclass, it is a callable object. Therefore, calling it with parentheses, e.g. with the M(classname, superclasses, attr_dict)
statement as we did above, will invoke its __call__
method. In other words, the class creation statements we presented above are translated into M.__call__("C", (B1, B2), attr_dict)
. Then, the default __call__
method provided by the type
(since it is the default metaclass of any metaclasses that don’t specify a custom one) invokes the __new__
and __init__
methods with the correct arguments. In other words, the following two methods are invoked from M.__call__
sequentially:
1. M.__new__(M, "C", (B1, B2), attr_dict)
Note that __new__
is a static method of M
. The first parameter is always the class whose instance was requested (e.g. by calling the constructor expression with M.__call__(…)
or M(…)
, or with usual class definition syntax). The remaining arguments are passed to the constructor expression to initialize the constructor parameters. The return value of __new__
is the new object instance, in this case a class object, C
.
Note: the signature of __new__
when defined inside a metaclass is as follows:
(mcls, clsname, superclasses, attr_dict) -> object
2. M.__init__(cls, 'C', (B1, B2), attr_dict)
where cls represents the new object created by __new__
in the previous step, which is the class object C
in our example.
The 1st method, __new__
creates and returns the new class C
, and the __init__
initializes it (sets its all class attributes, defines object methods and attributes).
Another Use of type
Although type
is the base class for all metaclasses in python (or the default metaclass for all the classes), there is another use of the type
keyword in python, which acts as a regular function. This type
function returns the type of its argument. In python3, class and type concepts are mostly used interchangeably. Each object has a class (or type) from which it was derived. When type
is called with an instance of a class (which is a plain object), it returns the class. Instead, if it is called with a class object, it will return the name of the metaclass of that class. In other words, the type (or class) of a class is the metaclass using which it was created. The following example can explain the type
function better.
We can see that the type of 5
is int
, which is a class whereas the type of int
is type
, which is the default metaclass. By the way, instead of calling the type
function, we can use the __class__
attribute to check the type of our object, which is available to all objects (including the classes or metaclasses since they are all objects as well.). To check the class of 5
, we have to use parentheses though, since the dot symbol defines a float value when it is following an integer.
The above examples show that the type and class is really the same.
Python has even a built-in mechanism used to verify the type of an object. This mechanism is the isinstance
function which returns True if its first argument is an instance of the second argument. Continuing our example, the following results should be expected.
This shows that 5
is an instance of int
, and int
is an instance of type
. Note that, here type
is the default metaclass, not the function presented in this section. While talking about the type
metaclass, let’s check its type (class) just for fun.
Oh, that was surprising, wasn’t it? Well, actually not very much. The output above means the type of the type
is the type
itself. Since type
is a class (actually a metaclass as it is used for instantiation of new classes), we need a metaclass to create it. It turns out that the metaclass to create the type
metaclass is the type
itself. We can verify that the class of type
is type
with the __class__
attribute as well.
Or, equivalently, the following statement verifies that type
is an instance of type
.
Inheritance Hierarchy vs Metaclass Hierarchy
Python’s metaclass mechanism design leads to some further interesting results. For example, let’s think of a custom metaclass which is directly derived from type
. Remember, to create a metaclass we have to extend type
, or any of its subclasses. The following is an example of such a custom metaclass.
Let’s check its type
(or __class__
attribute):
Okay, that was expected. What about its base classes (the classes it inherits from)? There is a __bases__
attribute in python that lets us query the direct parent (also called `super`) classes of a class. To clarify, when you have a class B derived from A, and a class C derived from B, if you call __bases__
on C, it will return only B, since it is the direct parent of C. Moreover, in the case of multiple inheritance, __bases__
will return all direct parents. Going back to the parents of our example metaclass:
The above statement says that the base class of the CustomMeta
is type
, which is its metaclass! In other words, CustomMeta
has to inherit from type
(to be a metaclass) and must use type
as its metaclass (to be instantiated) simultaneously. You can even go further while looking up CustomMeta
’s base classes to inspect the whole class hierarchy above it using the __mro__
attribute defined for all classes (including the metaclasses of course). __mro__
is an abbreviation for Method Resolution Order and it denotes the class hierarchy traversed when a method invoked on a class (or an instance of it). For example, if you call a method of a list
object, the name of the method will be searched in the __mro__
of the list
class in order. Similarly, if you call a class method of any class, the method will be searched in the __mro__
of the class again. Let’s check the __mro__
of our CustomMeta
class.
The output above shows that type
is indeed a superclass of the CustomMeta
. Note that, object
, being the base class of all python classes, will show itself in the __mro__
of any class in python. This includes the type
class as well. Moreover, the first item in __mro__
is the class calling __mro__
itself, which is CustomMeta
in this example. Let’s check the __mro__
of the type
:
This shows that type
has a single superclass, the object
class which means type
is directly derived from the object
. Remember, the first item in the MRO is the caller class itself (here, type
), so we do not take that into account as a superclass. Alternatively, we can directly use the __bases__
attributes to verify that object
is indeed the only superclass of type
:
These examples show that MRO depends on the inheritance hierarchy. Moreover, inheritance hierarchy is unrelated to the metaclasses. type
is the metaclass of itself, but is derived from the object
(just like any other class). Metaclasses are in fact real classes and have properties just as other plain classes. Similarly, classes are real objects and have properties just like any other objects. However, the converse of these statements is not true. An object does not have to be a class (i.e. class object). For example, __mro__
is only defined for classes (and also metaclasses of course) so regular objects do not have a __mro__
attribute. Similarly, a class does not have to be a metaclass. In fact, the only way a class can be a metaclass is if it inherits from type
, or a subclass of type
. In fact, the metaclasses form an inheritance hierarchy where type
sits on the top. A note for cautious readers is that object
is not in the metaclass hierarchy as it is not a metaclass although type
is derived from it.
Going back to the general inheritance hierarchy of the python classes, let’s check the bases and MRO of object
:
Well, as expected we get an empty tuple since object
is at the top of the python’s type hierarchy so it is not derived from any class. What about its MRO?
Remember, the first class in the__mro__
tuple of a class is the class itself. As object
does not inherit from any class, there is no other class in its Method Resolution Order.
Let’s also call the type function on object
to check its type (or the metaclass used to instantiate it) just for fun.
The above output shows that the object
class is an object that is instantiated from the metaclass type
. The following definitions are equivalent to this statement.
object
is an instance oftype
.
2. The class of object
is type
(technically type
is its metaclass, since object
is a class itself).
Again, the inheritance relationship is unrelated to the metaclass — class relationship. Metaclass — class relationship is actually a special case of class — object relationship such that the object that is being created is a class and the class used (as a template or blueprint) for creating that object is a metaclass.
The below figure also shows the object
and type
relationship in a visual way.
I refer the curious readers to [3] to read further about the type — object relationship and the class creation in C level.
An object instantiated from a class is conventionally referred to by the identifier self
. Therefore, the first parameter of the methods defined inside a class that are intended to operate on the generated object is named self
since they receive that object as their first argument. Similarly, the methods of a metaclass that are operating its instance (the class, represented by cls
) accept cls
as their first parameter to represent the class object. For example, the first argument of the __init__
in a metaclass is cls
by convention (although it does not have to), and the variables defined in __init__
as bound to cls
actually become class variables. These are just like the regular class variables we assign while defining a class statically and they will be accessible by the new objects instantiated from that class as well. Similarly, the regular methods defined in the metaclass will be class methods bound to the generated class, and the instances of that class can call them just like other class methods defined using the @classmethod
decorator.
Acknowledgement
Special thanks to Devrim Çavuşoğlu, Fatih Cagatay Akyon and Sinan Onur ALTINUÇ for their valuable feedback.