If you are using type hints in Python and static analysis tools such as mypy, then you have probably used the Any type quite often to get around typing functions and methods where the type of an argument can vary at runtime. While Any itself is a very convenient tool that enables gradual typing in Python code (the ability to add type hints to your code little by little over time), it doesn’t add any real value in terms of portraying knowledge to the reader. This is where generics come in. In this article I go over the basics of generics and how they can be used in Python to better document and communicate the intentions of your code, and in some cases even enforce correctness.
Why use type hints?
Before we start we should cover why we would use type hints in Python at all.
If you come from a background of dynamic languages, or maybe you are just starting out coding and have only worked on smaller projects, you may wonder what the value of adding type hints to Python code is. Isn’t the point of Python to be dynamic, rely on duck typing and enable the engineer to create something quickly without barriers? Well, yes, it is. But, if you work on a large codebase, or with other engineers, or even on a pet project that you only work on occasionally, you’ll find that the more information you can add to the code the easier it is to work on.
I’ve worked on some pretty large Python monoliths in the past where the only way to actually understand what was going on was to either get it running on your machine (often harder than it sounds) or to write a test that fires the code you want to understand, throw in some debugs and start stepping through it. Type hints add value as they give you more information about what is happening in your code so you can better reason about it without extra effort. These type hints also enable you to use tools like mypy that can check for errors that might have only been seen at runtime, as the type hints add a contract to your code that these tools can verify.
Type hints increase readability, code quality and allow you to find bugs and issues before your customers do.
What are generics?
The aim of generics are to:
- Allow functions, methods and classes to work with arguments of any type whilst maintaining the information on the relationships between things, such as arguments and return values.
- Better define how types can mix
We achieve these points by using generic types instead of concrete or parent types when defining what type is to be used in a given situation.
That may or may not have made any sense. The best way to explain this properly is with some simple examples. The next section will guide us through some examples of generics and how to use them in our Python code.
Using generics in Python
All of the following examples are written using Python 3.8 and therefore use the typing module.
Let’s say we have a function, and that function takes a list of things, and returns the first thing in the list.
The first() function defined here will work with a list of any type, even a list of mixed types. But, what happens when we decide to better define our lists? If we decide to limit the types of the elements in these lists, by adding type hints to our code and using mypy to enforce those types, you may end up with something like this;
This is a common sight when adding type hints. We better define the containers but we have just used Any on the first() function itself. This doesn’t add any value! In this very basic example, where we can see what is happening in the code at a glance, this isn’t too bad. However, even with this example, let alone a larger real world example working with user defined types, these type hints don’t capture something that we know to be true. We know that our method will be used with lists of a single type. We also know that the return type will match the type of the items in the list, so let’s capture these things in the code.
Here we have added a generic type named T. We did this by using the TypeVar factory, giving the name of the type we want to create and then capturing that in a variable named T. This is then used the same way as any other type is used in Python type hints. T and U are commonly used names in generics (T standing for Type and U standing for…. nothing. It’s just the next letter in the alphabet) similar to how i and x are used as iteration variables.
Using this generic type T, the first() function now states that the container parameter is a list of a “generic type”. We don’t care about the actual type of the argument, but we do care that the return value is the same type as the items in the list. Using this we capture the relationship between the argument and return value in code. It has the added bonus of allowing mypy to detect if we try to return something that is not from the container argument. Let’s see what happens if we return a value of the same type as the contents of container, but is not actually from container…
In the above example even though the only container argument passed to the function has elements of type str, and we return a str, mypy raises an “Incompatible return value type” error, as it was expecting a return value of generic type T .We only define T as the content type for the container parameter in this function, so the return value must come from the container.
Using generics in the first() function was a small change, but we now better communicate to the reader the relationship between the argument and the return value, and use that information to allow static analysis tools to check our code is correct.
Let’s use a few more simple examples to demonstrate everything we have just learnt to show us how useful this is. In these examples we use K and V as our generic types, as they represent the types of the keys and values of a dictionary.
Here, get_item() doesn’t care what types the keys of the dictionary are, or what type the values of the dictionary are, but it captures that the key argument we send, has to be of the same type as the keys in the container argument we send, and that the return value will be a dictionary value and not a dictionary key. We get all of this information just by looking at the signature of the function, and again this can be tested for correctness.
Let’s look at one final example:
Above, we use a poor name for our function but the generic types still explain it’s intent of returning the first key, not the first value. If we then switch the implementation of this to return the first value…
mypy raises the same “incompatible return value type” as we saw before, explaining what we have done wrong.
In these simple examples we have used generic types that can represent any type. However you can limit the types that can be represented by a generic type in Python. This can be done by listing the types that are allowed
Or by setting an “upper bound” type
This then limits what this generic type can represent to the upper bound type and subtypes of the given type, in this case int and it’s subtype bool.
Generic Types
Generics are not just used for function and method parameters. They can also be used to define classes that can contain, or work with, multiple types. These “generic types” allow us to state what type, or types, we want to work with for each instance when we instantiate the class.
Let’s look back at this earlier listing as an example:
Here we updated our code to use fixed type lists. We used the construct List[str] and List[int] to define that the list will contain only string and integer values respectively. This works because List is a generic type. When we instantiate an instance of a list, we tell it what type its values will be. If we did not define a type on instantiation, then it assumes Any. That means that my_list: List = ['a']and my_list: List[Any] = ['a'] are the same.
Most container types in the Typing module are generic, as they allow you to define what type the contents of the container will be on instantiation. In the case of Dict we can state the type of the key and value. Callable is another example of a generic type, as it allows us to define the types of the parameters as well as the return type.
To better understand the concept of generic types, let’s look at building one of our own
User defined generic types
In the following example we have made a registry class. The type of the contents of the registry is generic. This type is stated when we instantiate this class. After instantiation that instance will only accept arguments of that type.
Here we have created the generic class Registry. This is done by extending the Generic base class, and by defining the generic types we want to be valid within this class. In this case we define T(line 5) which is then used within methods of the Registry class.
When we instantiate family_name_reg, we state that it will only hold values of type string (by using Registry[str]), and the family_age_reg instance will only hold values of type integer (by using Registry[int]).
Generics have allowed us to create a class that can be used with multiple types, and then enforce (through the use of tools such as mypy) that only arguments of the specified type are sent to the methods of the instance.
Using our example above, if we try to set a string value in the age registry, we can see that mypy will raise this as an error
Generics are very powerful and help you to better reason about what you are doing and why. They also communicate that information to others, or your future self, that are reading your code and add a contract that static analysis tools can use to check your code is correct. In Python we don’t need to use them, but only in the same way we don’t need to add type hints. Much for the same reasons why we try to make code readable by writing small functions and using naming conventions that are meaningful, using type hints and generics just makes things easier to reason about, which means less time trying to understand what is going on and more time progressing your project.
In this article we have covered what generics are, why using them is a good thing and how to use them in Python. The examples were brief and simple, but I hope they portrayed how much value using generics adds to your codebase, empowers you to dive deeper into the subject and to start using them in your code!
