Manage instance attributes and its access to code more defensively in Python

No one needs intro to Python being an elegant language which many love, it is backed a great community of devs and it is widely used in different applications such as data, web, Desktop GUI, Business apps etc.

But there are some caveats, understanding how python’s inner machinery works will help us code better and more importantly write generic libraries which can be maintained easily.

This whole post in an inspired by reading one of David Beazley’s book on python and we used his recipes to write a framework on top of Apache Spark for creating maintainable applications. (ETL jobs using spark are easier to write but hard to maintain)

Understanding classes, instances and its attributes

We will start by understanding the basic concepts such classes, instances and attributes.

Since our use case had a lot of dependancies involved, I am taking a simple example which is easy for everyone to understand and will try to explain things. Let’s say we are creating a simple class in python and an instance of that class.

class Person(object):
    def __init__(self, name, age, address, salary):
        self.name = name
        self.age = age
        self.address = address
        self.salary = salary
person1 = Person("Erling", 22, 'US', 1000)

It seems we have a special attribute associated to the instance created called ‘__dict__’ which describes the object, meaning it contains all the attributes the object has.

person1.__dict__
{‘name’: ‘Erling’, ‘age’: 22, ‘address’: ‘US’, ‘salary’: 1000}

We can also access the attributes using the name of the attribute as a key to the dict attribute.

person1.__class__
<class '__main__.Person'>
person1.__class__.__dict__['name']
'Erling'

This is expected. But you might have faced this issue as a python programmer at least for once I am sure.

person1.salary = '2000' # wrong type
person1.adres = 'Norway' # wrong attribute name
person1.__dict__
{'name': 'Erling', 'age': 27, 'address': 'US', 'salary': '2000', 'adres': 'Norway'}

And now one of our attribute is of wrong type and because of a typo we accidentally created a new attribute, python didn’t complain and depending on the complexity of your implementation your program might go crazy and gets difficult to troubleshoot. This is well known since python is interpreted.

There are some ways to handle this, below is one of them.

Getters and Setters

Getters and setters work little differently in python when compared to other OO language (Java for example). The main objective to use it in python is may be to hide direct access to class attributes and probably add some custom validation or other logic when we access/modify an attribute of an instance.

Coming back to our case where we would want to avoid setting wrong type to an attribute, we can write code like below. (In python we use @property decorator, setter to achieve this).

class Person(object):
    def __init__(self, name, age, address, salary):
        self.name = name
        self.age = age
        self.address = address
        self._salary = salary
@property
    def salary(self):
        return self._salary
@salary.setter
    def salary(self, newsalary):
    if not isinstance(newsalary, int):
        raise TypeError('Salary expected an Int value')
    self._salary = newsalary
@property
    def age(self):
        return self._age
@age.setter
    def age(self, newage):
    if not isinstance(newage, int):
        raise TypeError('Age expected an Int value')
    if newage < 0:
        raise ValueError('Age should be more than zero')
    self._age = newage
>>> person1 = Person('Erling', 21, 'Norway', 1000)
person1.__dict__
{'name': 'Erling', '_age': 21, 'address': 'Norway', '_salary': 1000}

It is implied by the programmer that instance variables which starts with ‘_’ are inner implementations and it should not be accessed directly using instances. (python convention)

Now we can just access the age and salary as regular attributes which has our validation logic to it.

>>> person1.age = 25
>>> person1.salary = 2000
>>> person1.__dict__
{‘name’: ‘Erling’, ‘_age’: 25, ‘address’: ‘Norway’, ‘_salary’: 2000}
person2 = Person(‘Haaland’, ‘20’, ‘Norway’, 3000)
Traceback (most recent call last):
File “<stdin>”, line 1, in <module>
File “<stdin>”, line 4, in __init__
File “<stdin>”, line 21, in age
TypeError: Age expected an Int value

1. But this creates another problem of having too much boiler plate code and would get annoying soon. (Think about writing this every class that is created)
2. And still we haven’t handled the problem of adding new attributes in our instances.

Managing attributes by understanding the ‘.’

We know that we can get an attribute of an instance using the ‘.’ But how this works under the hood?

Lets say if we do

person1.salary = 2000

Seems Python would get the class of the instance, check its __dict__ attribute for the key ‘salary’, this would return a property, after that the property is checked if it has a ‘__set__’ method and then if it does the set method is called. An example to clear out the confusion.

person1.__class__
<class '__main__.Person'>
person1.__class__.__dict__['salary']
<property object at 0x10349af90>
person1.__class__.__dict__['salary']
<property object at 0x10349af90>
property = person1.__class__.__dict__['salary']
>>> hasattr(property, '__set__')
True
property.__set__(person1, 1000)

The same procedure applies for getting an attribute from an instance as well.

Now can we leverage this to reduce the lines of code we would need to write using property and setter for the classes we create? It seems we can.

class Integer(object):
    def __init__(self, name):
        self.name = name
    def __get__(self, instance, cls):
        return instance.__dict__[self.name]
    def __set__(self, instance, value):
        if not isinstance(value, int):
            raise TypeError('Expected an int value')
        instance.__dict__[self.name] = value

class String(object):
    def __init__(self, name):
        self.name = name
    def __get__(self, instance, cls):
        return instance.__dict__[self.name]
    def __set__(self, instance, value):
        if not isinstance(value, str):
            raise TypeError('Expected a string value')
        instance.__dict__[self.name] = value

In the above example I have created an Integer and String class which uses the inner machinery (using set and get methods) to intercept the ‘.’ when we access an attribute of an instance.

And now our code will look like this,

class Person(object):

    age = Integer('age')
    salary = Integer('salary')
    name = String('name')
    def __init__(self, name, age, address, salary):
        self.name = name
        self.age = age
        self.address = address
        self.salary = salary
person1 = Person("Erling", '27', 'US', 1000)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 7, in __init__
File "<stdin>", line 8, in __set__
TypeError: Expected an int value

The amount of code we wrote is greatly reduced to achieve the same functionality using the getter and setters. To further improve this code we will use inheritance to create a generic type class to reuse the get and set methods.

class Type(object):
    expected_type = object

    def __init__(self, name):
        self.name = name

    def __get__(self, instance, cls):
        return instance.__dict__[self.name]

    def __set__(self, instance, value):
        if not isinstance(value, int):
            raise TypeError('Expected{}'.format(self.expected_type))
        instance.__dict__[self.name] = value

class Integer(object):
    expected_type = int

class String(object):
    expected_type = str

class Float(object):
    expected_type = float
class Person(object):

    age = Integer()
    salary = Integer()
    name = String()
    def __init__(self, name, age, address, salary):
        self.name = name
        self.age = age
        self.address = address
        self.salary = salary

This really helped us when we were creating our framework, we can extend this functionality to have custom objects as expected_type, like a little type system if you will.

Neat and now to the final part of our problem, we need to restrict our instances to create new attributes.

It seems there is a more generic way of capturing getting and setting an attribute using the magic methods __getattr__ and __setattr__. When we call an attribute through an instance it is intercepted by __getattr__ and when we set an attribute __setattr__ method is called. We can override the __setattr__ method and add our check there.

person1 = Person('Rick', 65, 'US', 10)
>>> person1.age
65
>>> person1.name
‘Rick’
>>> person1.__setattr__(‘age’, 66)
>>>
>>> person1.__dict__
{‘name’: ‘Rick’, ‘age’: 66, ‘address’: ‘US’, ‘salary’: 10}

So now our Person class looks like this.

class Person(object):
    age = Integer()
    salary = Integer()
    name = String()
    def __init__(self, name, age, address, salary):
        self.name = name
        self.age = age
        self.address = address
        self.salary = salary
    def __setattr__(self, key, value):
        if key not in {'name', 'age', 'address', 'salary'}:
            raise AttributeError("No attribute {}".format(key))
>>> person1 = Person("Erling", 27, 'US', 1000)
>>> person1.age = 21
>>> person1.money = 1000
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 12, in __setattr__
AttributeError: No attribute money

And finally we can now restrict our instances to add more attributes. Hope these ideas help you in someway while creating libraries/framework in python to make the code more maintainable.

[1]: Python Cookbook: Recipes for Mastering Python 3 3rd Edition

https://www.amazon.in/Python-Cookbook-Recipes-Mastering-ebook/dp/B00DQV4GGY