TIQETS.ENGINEERING

Type Checking in Python

How we use it at Tiqets and what we love about it

Why do we use Python type checking at Tiqets?

Type hinting has made a visible difference in how we develop the Tiqets platform.

• Less cognitive load needed to work with our code. The types of parameters and return values are clearly stated and verified. No guessing, and no surprises.
• Realize mistakes early. If we return the wrong type, forget that a variable may be None, or accidentally redeclare a variable, the type checker will tell us.
• Data validation. We declare our data classes using attrs, which allows us to state the types of our attributes, as well as checking them at runtime. We may switch to Pydantic, which has better syntax.
• Avoid trivial unit tests. Type checking avoids the need to write and maintain trivial unit tests.

Overall, a better overall developer experience. As a scaleup, this is key for Tiqets’ rapid tech team growth.

What does it look like?

from typing import List
import attr
from attr.validators import instance_of
from pydantic import BaseModel
def sum_positives(numbers: List[int]) -> int:
    return sum(num for num in numbers if num > 0)
class Product:  # mypy will check this
    id: int
    title: str
class Venue(BaseModel):  # Pydantic will check at runtime
    id: int
    name: str
@attr.s
class Ticket:
    # attr will check at runtime
    id = attr.ib(type=int, validator=instance_of(int))

What types can I use?

In addition to the simple types (int, str, etc.) you can use the following types, defined on the typing module (introduced in Python 3.5).

• Collections: Tuple, Dict, MutableMapping, List, NamedTuple, etc.
• Composers: Union, Optional
• Callable
• Generics: TypeVar, Generic.

How do I get started with typing in my Python code?

Photo by Sapan Patel on Unsplash

You can get started gradually: if a function has no typing, it won’t be type-checked.

Here’s the step-to-step guide of how we did it at Tiqets:

1. Make the minimum changes needed to make mypy run without errors.
2. Add a mypy step to your CI build and pre-commit hooks. From now onwards, no new typing problems can be introduced. Things can only get better!
3. Add typing to all new code. Teach your team about typing and mypy
4. Gradually add typing to all existing code.

Our transition to typed Python

Typing pitfalls

With all its benefits, Python type checking is far from perfect. Here are our main dangers for which to watch out:

A false sense of security. Type checkers won’t catch all issues. Also, types aren’t checked at runtime unless you use a library such as attrs or Pydantic. Think of type checking as an extra safety step, not as a replacement.

No optimizations. Python won’t use the knowledge about types to make any optimizations on your code.

Untyped libraries will lead to typing mistakes. If your method returns the value of a function that doesn’t have typing, you must manually verify that the function you’re calling returns the type you specified.

Type hints can get ugly, for example Dict[str, Union[str, Union[int, bool, Venue]]]. Our tip on that is: if your type is complex, you may be using the wrong type and should fix that.

Beyond the basics

Photo by Francesco Ungaro on Unsplash

Define your types

You can define a type by using TypeVar. An example of where this is useful is a function that returns an element from a sequence.

from typing import Sequence, TypeVar
T = TypeVar('T')      # Declare type variable
def first(l: Sequence[T]) -> T:   # Generic function
    return l[0]

Type hinting classmethod constructors

TypeVar is also useful when declaring the types of a classmethod constructor.

On the example below, BaseModel.from_dict returns a BaseModel, and on the subclass Product(BaseModel), Product.from_dict returns a Product. The type T must be either BaseModel or a subclass of it.

Here’s what the inheritance looks like:

object ⤑ BaseModel ⤑ Product

The parameter bound='BaseModel' sets BaseModel as the upper boundary of the type: it can be BaseModel or a subclass of it, but it cannot be less than BaseModel (i.e, object).

from typing import TypeVar, Dict, Any, Type
T = TypeVar('T', bound='BaseModel')
class BaseModel:
    def __init__(self, id: int):
        self.id = id
@classmethod
    def from_dict(cls: Type[T], values: Dict[str, Any]) -> T:
        return cls(**values)
class Product(BaseModel):
    def __init__(self, id: int, title: str):
        super().__init__(id=id)
        self.title = title
product = Product.from_dict({"id": 1, "title": "Great Product"})

Why bound='BaseModel' instead of bound=BaseModel? Because BaseModel hasn’t been defined yet when we create the TypeVar. Not a fan? We neither—you may want to enable postponed evaluation of annotations (see PEP 563).

The example below will pass the type checking, but it will fail at runtime, because BaseModel needs to be defined when TypeVar is called. This is an example of a case in which type checking won’t catch an issue.

from __future__ import annotations
from typing import TypeVar, Dict, Any, Type
T = TypeVar('T', bound=BaseModel)
...

Detect unsafe input

If you receive unsafe strings from your users, you may want to define a new type for them. The type checker will then verify that you’re not sending unsafe strings to functions that accept only safe strings.

from typing import TypeVar
UserProvidedStr = TypeVar('UserProvidedStr')
def do_something(value: str) -> None:
    pass
def sanitize(value: UserProvidedStr) -> str:
    return 'Sanitized value'
def handle_request(unsafe: UserProvidedStr, safe: str):
    do_something(unsafe)  # error: Argument 1 to "do_something" has incompatible type "UserProvidedStr"; expected "str"
    do_something(safe)    # This works
    do_something(sanitize(unsafe))  # This works

Not just types: Literals

Python typing is not just about types. Take open for example:

• If mode is "r" , it will read text
• If mode is "rb", it will read bytes

You can make this dependency between parameter value and type by using a Literal.

from typing import Literal
@overload
def open(fn: str, mode: Literal['r', 'w']) -> IO[Text]: ...
@overload
def open(fn: str, mode: Literal['rb', 'rw') -> IO[bytes]: ...
def open(fn: str, mode: str):
	# Your implementation goes here

Typed dictionaries

When you need a typed dictionary, consider if a data class wouldn’t serve you better. Still, you can use typed dictionaries from Python 3.8 onwards.

from typing import TypedDict
class ProductDict(TypedDict):
    id: int
    name: str
product: ProductDict = {'id': 1, 'name': 'Best Product'}
# This won't pass the check
broken_product: ProductDict = {'id': '1', 'name': 'Best Product'}

Final classes, methods, attributes, and variables

On Python 3.8, you can define classes, methods, and variables to be final.

• A final class can’t be subclassed.
• A final method can’t be overloaded.
• A final variable can’t be reassigned.

from typing import final, Final
@final
class Base:
    pass
class Derived(Base):  # error: Cannot inherit from final class "Base"
    pass
ID: Final = 3
ID = 4  # error: Cannot assign to final name "ID"

Sphynx

Use sphynx-autodoc-typehints to allow Sphynx to use the types you defined when it generates the documentation.

Trickier type hints

Photo by Patrick Robert Doyle on Unsplash

What about duck typing?

Duck typing can be type-checked too. You can explicitly define how your function needs its parameters to quack, and the type checker will make sure they do.

Imagine a function that closes things such as connections or files. This function assumes the object passed as a parameter to have a close method that takes no parameters and returns nothing. You can make this assumption explicit by defining a Protocol.

Below, we create the Closeable protocol: any object that has a methodclose that takes no parameters and returns nothing is closeable. These objects are not aware of the protocol.

from typing import Protocol, TypeVar
class Connection:
    def close(self) -> None:
        pass
class Bug:
    def close(self, user_id: int) -> None:
        pass
# Connection and Bug aren't aware of the protocol below.
class Closeable(Protocol):
    def close(self) -> None: ...
def do_close(c: Closeable) -> None:
    c.close()
# This is ok
do_close(Connection())
# This will fail mypy check
do_close(Bug())
# error: Argument 1 to "do_close" has incompatible type "Bug";
#        expected "Closeable"
# note: Following member(s) of "Bug" have conflicts:
# note:     Expected:
# note:         def close(self) -> None
# note:     Got:
# note:         def close(self, id: int) -> None

Generics

Container classes can also be type-checked. To do that, we must define a new type that represents the type that the class contains. Below, we define a type T for our container. When it contains a number, for example Container(123) , T will be int; when it contains a string T will be str.

from typing import TypeVar, Dict, Any, Type, Generic
T = TypeVar('T')
class Container(Generic[T]):
    def __init__(self, value: T):
        self._value = value
def get(self) -> T:
        return self._value
def read_int_container(c: Container[int]) -> int:
    return c.get()
def read_str_container(c: Container[str]) -> str:
    return c.get()
# This works:
read_int_container(Container(123))
read_str_container(Container("hello"))
# This won't pass the mypy check
# error: Argument 1 to "Container" has incompatible type "str"; expected "int"
read_int_container(Container("hello"))

Multiple return types

What if your function returns different types depending on the type of its input parameter? A simple but wrong approach would be:

from typing import Union
def double(value: Union[int, str]) -> Union[int, str]:
    return value * 2
reveal_type(double("a"))  # The type is Union[int, str]
reveal_type(double(1))    # The type is Union[int, str]

While it is true that double returns int or str, that doesn’t capture the whole truth: this function returns int when its input parameter is int, and str when the parameter is str.

The proper way to define the type of double has the downside of being a bit verbose.

from typing import Union, overload
def double(value: Union[int, str]) -> Union[int, str]:
    return value * 2
@overload
def better_double(value: int) -> int:
    pass
@overload
def better_double(value: str) -> str:
    pass
def better_double(value):
    return value * 2
reveal_type(better_double("a"))  # The type is str
reveal_type(better_double(1))    # The type is int

Solving type lookup issues

The type checker will look for the type in the closest namespace. On the following example, mypy will think that the method returns values of type A.float, where we really mean it to return the builtin float.

class A:
    def float(self) -> float:
        return 1.0

You’ll have to state explicitly that you mean builtins.float.

import builtins
class A:
    def float(self) -> builtins.float:
        return 1.0

Typecasting

If despite your best efforts you can’t get the type checker to correctly infer your type, you can force a type using cast.

from typing import cast, List
value = [130]
value_float = cast(List[float], value)
reveal_type(value_float)  # The type is inferred as List[float]
value_float == [130]      # But this is still true

Be careful withcast: as you can see, it’s easy to introduce hard-to-track-down bugs.

Giving up: Ignoring errors

If nothing else works, you can add a comment # type: ignore to a line to avoid it being type-checked.

The future of typing in Python

Photo by Natalia Y on Unsplash

Python type checking is alive and keeps improving. Here are some features to which we are looking forward:

Unified error codes throughout all type checkers. This will allow tools such as editors to easily interpret the typing errors independently from which type checker you’re using.

Less camel case and imports: use list[] instead of typing.List[int].

Shorter syntax

• int | float instead of Union[int, float]
• ?str = "" instead of Optional[str]

What type checker do you use at Tiqets?

We use mypy, which is the reference implementation, and was the best tool when we started using typing (back then, our codebase was still Python 2.7).

There are a few interesting alternative implementations:

• pyright (Microsoft) is faster but runs on nodejs instead of Python.
• pytype (Google) can infer types from unannotated code. For more details, check out this comparison between mypy and pytype.
• pyre (Facebook) is able to do incremental checking.

Where should you add type hints?

We add type hints to all our code.

You can think of type hints as a particular form of unit tests. They allow you to automatically test the inputs and outputs of your code. And because they are shorter and right in the code to which they hint, they are easier to maintain than unit tests.

Consider adding type hints everywhere you would add unit tests.

Author

Òscar Vilaplana is a software engineer at Tiqets. He maintains the Tiqets Tech Blog and writes fiction in his free time.