Python: Under the hood [Memory Allocation & Management]
Hi, in this article, We’ll be learning about some core concepts of python without getting into coding.
Firstly, why do we even care about under the hood stuffs when they are just meant to be “under the hood,” you may ask, right? Well, being a developer, it’s useful to know how a programming language works. As it could unravel the mystery of magical stuff happening in the background you don’t know of.
And for some reason, if your consciousness is telling you, that this is going to be challenging. Trust me it’s not. These are relatively simple concepts, just like the Python language itself.
Here is the list of the topics we are going to cover in this article series.
- Memory Allocation & Management in Python
- Garbage Collection
- GIL (Global Interpreter Lock)
Memory Allocation & Management in Python
Knowing about memory management may help you write efficient code. Even though you may not have control over memory allocation, but you can optimize your programs to allocate it better.
Before we dig deep, remember this:
In python, everything is object.
Languages like C, C++, or Java store the value in memory, and there’s a variable that points to that memory location.
Whereas In python, an entire object is stored in the memory (heap memory); the object could be integer, string, or a list, and variable pointing to an object. Confusing. Right?
To make sense, let’s understand what does a python object holds!
A PyObject in memory holds
- Type: integer, string, float, etc
- Ref Count: number of references bound to that object
- Value: value/data/information
So, whenever you create a variable, let’s say a = 200, a new PyObject is created in the memory, and its ref count is set to 1, and variable “a” points to it.
Okay. But what is ref count?
Let’s take an example to understand it.
We have a variable “a” of type integer with value of 200. Let’s say I need to have another variable named “b” of type integer with a value of 200.
So, you’ve created two variables like this.
a=200
b=200
Now, you might be guessing; there must be two objects in the memory for variable “a” and “b”. But that’s not the case. Both “a” and “b” points to the same object.
Let’s validate this through code.
As you can see, variable “a” and “b” have the same id (memory location) thus, representing the same object, while c has a different value of the id. Therefore, currently in memory, there two python objects.
So, what will happen if we assign a new value to “a” i.e., a = 3.
“a” now points to a new object, but “b” stills points to the same object.
But let’s consider this example.
In this example, a =1 but “b” has the value of “a”. When we change the value of “a”, is it going to affect “b” too? Let’s see.
“a” now points to a new object, but “b” still points to the older object. Quite Surprising. But why?
Because “b” does not directly point to the variable “a” but the object of variable “a”. That’s what ref count does. It keeps track of the number of variables pointing to it.
Another intriguing question would be, What would happen if the ref count of a PyObject becomes zero?
Before that, let’s find out how we can remove references to an object.
- del
Python’s built-in del keyword helps us remove these references pointing to an object. In fact, some people think that del removes the object from memory, which is not the case.
So, you could use del like this to remove a reference.
2. Going out of scope
The reference to an object gets automatically deleted once it goes out of scope.
Let’s take the above code as an example, the object of variable “x” will have +1 ref count inside the scope() function, but as the interpreter leaves this function scope, the ref count of the PyObject reduce to -1 because “x” is a local variable and only have scope to its function.
And this is one of the reasons why python discourages using global variables since variables in global scope stick to the entirety of the program.
3. Assigning a new object to a variable.
When assigning a new object to an already existing variable, the ref count of the previous object reduces to -1.
Now, coming back to the previous question, What happens when the ref count of an object is 0. Does it stay in memory?
Well, the answer is No.
As soon as ref count of an object turns to 0. It gets erased from the memory by Garbage Collector. You can think of garbage collector as a cleaner, which cleans all the leftovers and waste items which won’t be used in the future and this technique is called “Garbage Collection”.
We’ll be discussing garbage collection in python in more detail in the next article.
Note: If you have any doubts or suggestions, I’ll be happy to reply through the comments, and I’ll also be needing those lovely claps to boost myself to write more articles, so please be sure to do that.
