Photo by Alexander Andrews on Unsplash

Instance Creation in Python

Understanding the internals of Instance Creation in Python

In this article, we’ll take a deep dive into exactly what happens when we instantiate a new object in Python. With this knowledge in hand, we can exercise fine control over instance creation, which can allow us to customize Python objects in powerful ways. So let’s get started.

So what happens behind the scenes when we create a new object in Python?

Let’s try to understand this using a basic class example that models Employee entity.

class Employee:
    def __init__(self, name, age):
        print(type(self), self.__dict__)    # debug stat.
        self.name = name
        print(self.__dict__)                # debug stat.
        self.age = age
        print(self.__dict__)                # debug stat.

The above-defined constructor for our Employee class accepts the name and age attributes. We’ve added a few debug prints to monitor the state of the instance dictionary as assignment statements are executed. Let’s create an object of the Employee class and observe the results:
NOTE: Python internally stores the object attributes as a dictionary called dunder dict (__dict__). This dunder dict is just another normal dictionary.

>>> p = Employee('Sarah', 27)
<class '__main__.Employee'> {}
{'name': 'Sarah'}
{'name': 'Sarah', 'age': 27}

If we observe closely, we’ll see that the self already has the required type i.e <class ‘__main__.Employee'>. The instance dictionary is initially empty but as we continue to advance, the dictionary is populated by assignments to attributes of self.

One thing to note is the dunder init method does not return anything. It simply changes (or mutates) the instance it has been given.

Therefore, we can conclusively say the dunder init is responsible for initializing the instance it has already been given.

Now the obvious question is if not dunder init then who is responsible for creating the instance?

If we look at the special methods of Employee class using the built-in dir method, we can see one called dunder new __new__.

>>> dir(p)
['__class__', '__ne__', '__new__', ... , 'age', 'name']

We haven’t defined dunder new in our Employee class but we do inherit an implementation from the universal base class object. So it is the base class implementation of dunder new which is responsible for allocating our object in this case.

>>> p.__new__
<built-in method __new__ of type object at 0x10e003bc0>
>>> p.__new__ is object.__new__
True

The above snippet confirms that the dunder new is in fact the very same method as object new.

Hence, inherited __new__() allocates the object which is passed to the __init__() constructor as self.

Now, let’s try to override dunder new method to understand its signature better. We’ll implement the most basic override of dunder new, which simply delegates to base class implementation, along with adding few print statements, to inspect the arguments and return value.

class Employee:
    def __new__(cls, *args, **kwargs):
        print("args=", repr(args))
        print("kwargs=", repr(kwargs))
        obj = super().__new__(cls)     # same as object.__new__(cls)
        print("id(obj)=", id(obj))
        return obj
    def __init__(self, name, age):
        print("id(self)=", id(self))
        self.name = name
        self.age = age

The first thing to note is, dunder new is a class method. It expects cls as its first argument rather than self. The cls argument is the class of the new object, which will be allocated. In addition, dunder new accepts whatever parameters have been passed to the constructor. In this case, we have used the *args and **kwargs for the same.

The main purpose of dunder new is to allocate the instance of the calling class. All object allocation must be done by the dunder new implementation on the ultimate base class object. We’ve used super() to refer the object new method since that is more maintainable. (if ultimate base class object changes in future).

Finally, we are returning the newly created object obj which is passed to the __init__ method as self. This can be confirmed by comparing the ids of both objects.

>>> p = Employee('Sarah', 27)
args= ('Sarah', 27)
kwargs= {}
id(obj)= 4473868880
id(self)= 4473868880

From the above snippet, we see that the id of obj in dunder new is equal to the id of self in dunder init, and also that the arguments have been forwarded to dunder new as expected.

Hence the overall mechanics of the object allocation can be summed in the below figure.

Steps of object allocation

Summary:

We’ve covered the distinction between the allocation and initialization of instances.

• dunder new __new__() allocates and returns new instances. It is an implicit class method that accepts class of the new instance as its first argument.
• object.__new__() is the ultimate allocator which allocates all instances.
• dunder init __init__ is responsible for mutating the instance it has been given.