Authored by David Euresti
Use attrs. .. for everything. Even strongly typed code!
Here at Pilot we love love love Python. We use Twisted on Python 3 for our backend and like Glyph, we use attrs for everything. I haven’t written an __init__ method in over four years!
As many of you know Python is a dynamically typed language. However since we were retrieving data from many data sources (Quickbooks, Stripe and other integrations) we wanted to have more assurances that our data model was correct. So four years ago, when Pilot started, we leveraged the power of attrs validators to do type checking. First we had a function called typed that returned attr.ibs with InstanceOf validators attached. Eventually, since we were using Python 3, we moved to using auto_attribs and we wrote our own decorator that used the type annotations to check the types in a __post_attrs_init__. Our code now looked like this:
Then I thought, “Hey, we can probably use mypy to type check our code.” Sadly, four years ago me was wrong. Mypy didn’t support attrs and so didn’t know about the __init__ methods that it generated. Running mypy on our code base would yield lots of “Too many arguments for “ errors. I eventually saw the mypy issue about attrs and a solution was suggested, “add attrs support to mypy using a plugin.”
Over the next couple of weeks I started coding. First I had to create the entry points for the plugin. attrs works as a class decorator and back then mypy plugins had only been used for function decorators. Once I had that done I had to actually write the plugin. As it turns out attrs is incredibly complex and has a lot of options. I basically had to walk the entire class definition looking for “assignments” that look like attrs. It took a long long time to get it done and lots of tests. Hilariously, during that time attrs changed how they traversed the MRO (Method Resolution Order), fixing a long standing issue, and I had to change my code to do it the new way.
Finally I was done. And I ran it on our code. And it worked. Then I ran it a second time, and it didn’t. It turns out that mypy has a cache it stores on disk to make things faster and I wasn’t storing anything in the cache. So I had to go learn about mypy’s serialization methods and implement handlers that converted my data structures to JSON so they could be saved into the mypy cache.
I added support for __cmp__ and for frozen and for so many things. All the while thinking, I hope this will be useful. Eventually after 84 commits and a couple hundred messages back and forth it got merged. I also have to give props to chadrik who worked on the typeshed stubs which helped a lot.
Then I finally got to run it in our code. Thankfully because we had that @type_checked decorator pretty much all our classes had good __init__ annotations. However since this code had never been type checked before it had lots of errors. So to add mypy to our CI we used a big blacklist:
We chose to blacklist rather than whitelist because we wanted all new code to be good and we didn’t want to update a whitelist every time. Several python 2 projects do a grep for # type but in Python 3 you can’t really use that. It was more useful to just run mypy on the codebase once, blacklist those files and then you’d basically stop the bleeding on every other file.
There were a total of 432 mypy errors across 74 files. All these files were added to the blacklist. I filed bugs for all of them and people would pick up the bugs and fix the type errors. In a couple of months we’ve whittled that list down to 6 files.
I chose to start with strict_optional=True because I knew migrating after the fact would be a pain. warn_redundant_casts and warn_unused_ignores help you clean up “mypy silencing” after the particular issue has been fixed in mypy or typeshed.
We then started seeing another problem. Partially typed methods. Because in Python 3 the annotations are on each variable it was common to see methods like this:
Mypy would check these methods but it would use Any for those variables. And when you have an Any you don’t really have type checking. So we eventually turned on disallow_untyped_defs=True with a pretty large blacklist. We blacklisted all our tests since those can get pretty hairy. But apart from that we have about 100 files in which we disable this and we’re slowly whittling those away too.
Four years later we are quite happy with how we have whittled away at the blacklists. We’re not at zero but we’ve pretty much decided that we don’t need to worry about the files that are left.
So what kind of errors does mypy catch? My favorite error caught was
The comma was superfluous but it basically made the statement if self.variable always True. Testing could catch this but mypy will catch an error like this without needing to write tests.
So, in conclusion:
- Attrs is great! It makes writing python so much easier.
- Mypy is great! It makes catching errors so much easier.
- Combining both is amazing! 2 great tastes, that taste great together!
References:
Glyph’s post: https://glyph.twistedmatrix.com/2016/08/attrs.html
Mypy: https://mypy.readthedocs.io/en/latest/introduction.html
Mypy Issue about attrs: python/mypy#2088
PR Implementing attrs plugin: python/mypy#4397
