Static Type Checking in Python: Where Did the Ducks Go?

Exploring what makes type checking in Python so awesome, why its benefits are not only limited to safety, and how we are employing these tools in Systematic Fixed Income.

Where Did the Ducks Go?

We all love Python for its numerous qualities: it is extremely versatile, flexible, enjoyable to write, and has a vast package ecosystem. No wonder it has become the de-facto standard in several domains. But whilst its flexibility makes Python ideal for prototyping, it also makes the development and maintenance of large libraries and platforms more cumbersome. In other languages, typing proved to come to the rescue in such cases; hence, now, Python features static type checking. In this post, we will explore what makes type checking so awesome, why its benefits are not only limited to safety, and how we are employing these tools across BlackRock. If not curious about any of these, stay tuned to find out where the ducks went 😉.

What is it?

Sprinkle your code with type annotations and verify your program with a type checker such as the popular mypy. Even though Python remains a dynamically typed language, adding type hints and checking them statically combines the strengths of compiled languages with Python’s native flexibility. Type checkers are used like linters, with the type annotations being ignored at runtime.

The syntax is pretty straightforward. In this example, my_sum takes a list of integers and a boolean as inputs and returns an integer:

Python provides a powerful and easy-to-use type system with modern features such as type inference, union types, generics, and both structural and nominal subtyping while allowing you to escape back to dynamic typing if needed. Typing makes Python programs easier to understand, debug and maintain. The syntax is supported from Python 3.5, while for earlier versions one can add annotations as inline comments or separate stub files (each .py file is paired with a .pyi file annotating all signatures in the .py file).

Why use it?

Because it makes life easier. The primary goal is to make programs more robust by catching both common and hard-to-find errors early. However, typing also enhances the overall programming experience through IDE auto-completion and real-time checking, self-documenting code, the avoidance of boilerplate tests, and transferable concepts from other languages. Check the ‘Pros’ section towards the end for more.

Because it is becoming normal. Static type checking has been around for a few years now and is becoming the norm. Tech giants such as Dropbox, Facebook, Google, and Quora have been leaders in developing the tools and making use of the benefits. For example, Dropbox type checked over 4 million lines of Python code. All these firms also implemented their own type checkers, Dropbox’s mypy and Facebook’s pyre being the most popular. Significant open source projects are also leading the way, including Airflow (the largest Apache Python project), FastAPI, Prefect, and the async-io libraries.

Because it is trendy. Since 2015, type checking has been a very hot topic at Python conferences: it appeared in 10 PyCon lectures, 6 Europython and dozens of local PyCons. The trend is not limited to Python. Initially, Flow enriched JavaScript with types; this has ultimately evolved into TypeScript which has gained huge momentum in the JavaScript community. PHP also added type hints, while most of Facebook is powered by Hack, another typed PHP dialect. Ruby also features gradual typing through Sorbet. Typing is taken seriously by most new “cool kids”: Scala, Kotlin, Dart, Rust, and by the most hipster-ish ones, including Nim, Pony and Clojure.

Because Guido said so. The mypy project was started by Jukka Lehtosalo who was later joined by Guido van Rossum himself, Python’s “inventor” and former BDFL.

When to use it?

In his excellent article “The state of Type Hints in Python”, Bernat Gabor recommends that “Type hints should be used whenever unit tests are worth writing”, realpython.com reinforcing his statement.

  • This is very easy to do for new projects: 100% annotated from day 1 → type check in CI → profit” as advised in so many PyCons.
  • When dealing with large existing codebases the advantage is that Python allows for gradual typing. We recommend typing and checking the backbone of the codebase to ensure that the project is built on solid foundations, and then to contribute typed code going forward. When writing a library it is nice to also annotate user-facing functions for IDE auto-completion and type checking on their end. Tools like Pyannotate or MonkeyType can even be used to auto-annotate code. Having a type checker like mypy in CI is useful even without annotations since it checks the existence of all classes and functions referenced in imports.

How to use it?

Life before type annotations

Let’s think about a very common task: submitting jobs. Each job has an id and can be submitted. In “old-style” Python one would write code such as the below. To be good citizens, in real-life we must also implement methods like __eq__ and __repr__.

Now let’s see how we would write a function to submit some jobs.

As you can see, it is hard to tell what sort of jobs and what sort of iterable submit_jobstakes. In this case, it is obvious that jobs cannot be a generator since it is not sized but, in real scenarios it is hard to work backwards. The code above exposes some potentially dangerous behaviour: partial job submission.

Life after type annotations

In “modern” Python, we can use typing to make our example much more readable. In the following, we use Literals to say that some attribute takes values only within a finite set. We are also using dataclasses which reduce the need for boilerplate code like __init__, __eq__, __repr__, but mandate typing.

One important thing to note is that this example would trigger a type checker error but would not raise a runtime exception if one passed status st4. The elegance of the dataclass/Literal syntax comes with the cost of reliance on our type checking tools. If one wants to validate the arguments at runtime, however, one can add another decorator to the dataclass: @pydantic.validate_arguments(config=dict(arbitrary_types_allowed=True)) Pydantic is a library that allows runtime type checking based on type annotations. The same pydantic decorator can be applied to functions/methods as well.

Life with type annotations

Now, back to submitting jobs. We can safely tell what sort of jobs and what type of iterable are expected. In the following comments, we will use “GOOD” to mean “type checks successfully” and “BAD” to mean “static type error”.

On top of mypy and pyre (the main checkers), PyCharm provides a built-in type checker which provides feedback as you type — with no need to run anything. However, its functionality is limited. For example, in the example above mixed types are not flagged as wrong even though they are clearly violating our type declarations. Nevertheless, it’s a pretty useful feature, and capable of flagging a number of issues, and it keeps improving. We use GOOD/BAD for what mypy agrees/disagrees with, since it is by far the more specialized tool.

Union Types

Let’s assume that we want to use the same submit_jobsfunction on both types of jobs. This is when Union types come in handy. Union[MyJob, YourJob] means that it can either be MyJobor YourJob. The code that you write must cater for both types or explicitly handle special cases.

Observe that mixed type lists are allowed. For example, if we want to disallow mixed type lists we turn to type variables. Type hinting in Python is flexible and, as we will show, powerful.

Type Variables

Type variables mean that they can be substituted by any “concrete” type, potentially with some condition applied. For example, below List[TJob] can be either List[MyJob] or List[YourJob], the type checker will infer which one.

To illustrate how type variables are bound let’s take a different example: returning a random element from a sequence/list. The takeaway here is that we cannot have an output type var that doesn’t appear somewhere in the inputs. We’ll see later how to use Generics to overcome this.

“reveal_type” is a mypy function that does not exist in Python. That is, it would print out the types when running mypy to gain some insight into what mypy sees but needs to be deleted before running the Python code.

Where did the ducks go? 🦆➔ Structural subtyping a.k.a. Protocols

Duck typing is great for prototyping but pretty fragile in large systems. However, Pythonistas like the simplicity of duck typing. Fortunately, structural subtyping comes to the rescue. It is similar to Java interfaces with the exception that classes don’t “implement” the interface explicitly. Going back to our jobs example: what if, in the future, we may add HisJob, HerJob etc? If we were to use a Union type we would need to update it like Union[MyJob, YourJob, HisJob, HerJob]. This is clearly hard to maintain, annoying and not Pythonic. What we want to say is: as long as it has a field/attribute called “id” that is hashable (e.g. int, str, etc) and a method called submit() we can pass it to our submit_jobs function. We can use a Protocol (a.k.a interface) to make duck typing type-safe.

Protocols do not necessarily need to be generic (i.e. depend on a type variable like H). If you are curious about how runtime_checkableallows isinstancechecks without inheritance, metaclasses come into play ( __instancecheck__, __subclasscheck__, __subclasshook__). Just FYI, subtyping via inheritance is called nominal subtyping while subclassing via structure is called structural subtyping. Python “protocols” may be seen as a parallel for C++ “concepts”, TypeScript “interfaces”, Scala/Haskell “type classes” etc.

More Generics, Overloads, Forward references

Recap:

  • Output type variables must be bound by function input variables.
  • When using Union types we mean any type of the union is equally possible.

In the example below we build a custom list. We show how Generic[T] binds the type variable to the whole class. That is, all the Ts in this class must be the same and consequently we can return Ts without having Ts as inputs (since T is bound at class level rather than function level like S). Furthermore, we use overload to specify that __getitem__ either returns a T if given an int or a MyList[T] if given a slice like [:2]. Overloads basically rule out slice -> T and int -> MyList[T] which would be considered if annotating only with Unions. We also use forward references (in quotes) for the first time and outline the pattern for factory methods like “empty”.

To learn more, curious readers can go ahead and read about covariant and contravariant type vars (or functors in general). Even though out of scope, if you want to find out where monads and applicative functors fit in keep an eye on projects like dry-python.

Pros

  • Robustness: Type checking is a means of testing so it makes code more robust and eliminates the need for boilerplate tests, thus allowing engineers to focus on testing the logic rather than exceptions. Test coverage automatically becomes a more meaningful metric too since tests are on point.
  • Rich user experience: IDEs use annotations for autocompletion and refactoring purposes. Typed Python is self-documented so more readable and concise. Docstrings become smaller focusing on the logic and can no longer go stale.
  • Reveals code smells early on: Very complex types such as “List[Tuple[Job, Dict[str, int], Dict[Tuple[bool, bool], int]]]” indicate that the code is not quite right → refactor sooner rather than later.
  • Doesn’t get old: Type safety comes from sound theoretical models that have been researched and used for years.
  • The buzzwords: Extensible — mature ecosystem of stub files and plugins, Transferable — typing syntax is very similar to so many other programming languages, Scalable — “dmypy” daemon is designed to work on really large codebases, Gradual — typed and untyped code can coexist. Accessible — mypy is installed via pip.
  • Makes code faster: With the addition of mypyc one can compile annotated Python modules into C extensions, hence improving performance considerably.

Cons

On the flip side, one would need to learn a bit of new Python syntax and numerical packages like Pandas have limited support by default and initiatives like Pandera aim to close this gap. Numpy is riding the wave by actively adding type stubs. Research suggests that static type checking greatly improves code quality but it does not completely spare us from bad code; therefore extensive testing, linting and peer reviews are still to be used in conjunction.

We have also identified a few limitations of the typing module, like missing variadic type variables (compared to variadic templates in C++) or higher-kinded types (HKTs) but fortunately these features are being incorporated either in the Python core library or in projects like dry-python. In the interim, within our team we wrote stubs and a mypy plugin to unlock the full type checking potential on our libraries.

How does it work for us?

As a team we have been early advocates of type hinting within BlackRock, and feel it is time to share the hugely positive experience we have had. At the time of writing, we already have tens of thousands of typed Python code lines. The Pros above are mainly drawn from our personal experiences with this technique and what we learned from it.

Type checking has been particularly useful in places where high unit-test coverage is difficult to achieve, such as web-scraping or alternative data ETL processes. Moreover, type checking helped us fail early: once we ship something it is fairly uncommon to revisit it unless a system that we depend on changed. Adding new features is also easier. Type checking has a snowball effect: the more annotations, the easier it is to extend a system without breaking it. Since our team works on quite a few libraries at the same time, developers and users are able to benefit from the corresponding speed increase through IDE auto-completion and auto-documentation. This means support requirements are significantly reduced, freeing up time for both developers and end users.

Appendix — Simplified syntax

In Python 3.10 the typing syntax received a face-lift, but through a simple import from __future__ import annotations we can benefit from the same syntax starting from Python 3.7. That is, instead of importing “shadow” classes for built-in classes such as list, tuple, dict we can use them directly. Similarly, union types can be defined with via “pipe syntax” just like in TypeScript. For example:

Note that new style syntax does not currently play well with pydantic runtime validation.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
BlackRockEngineering

Official BlackRock Engineering Blog. From the designers & developers of industry-leading platform Aladdin®. Important disclosures: http://bit.ly/17XHCyc