Industrialized python code with Pylint plugins in Pants

Photo by Museums Victoria on Unsplash

Along with the enablement of Pants Build in one of our python multi-project repositories, at Doctrine, we are considerably strengthening our code conventions. What is Pants? A build system that can manage a repository containing several projects, in several languages and with various life cycles.

How do we come up with code conventions? Apply automatically the standard of Python, PEP8, and Black for instance, then for everything that’s a company convention, build custom checkers with Pylint, a.k.a. 1st-party plugins.

A vast amount of documentation for custom checkers (and Pylint plugins in general) in Pylint is available online, and we’ve selected a few very good ones:

• the official Pylint doc!
• a fully furnished doc from Oppia

Back to our multi-project repository, how should we build custom checkers with Pants? One would start with what Pants documentation suggests on the Pylint configuration page. This describes how to set up your custom plugins, and how to call them while calling ./pants lint .

There are a few items you need to understand in this doc:

• ✔️ the plugins folder needs to have its own target; as mentioned, it’s helpful to have a dedicated resolve. So let’s have a dedicated project. That’s a lot better since the code it embeds is supposed to be run only during CI, so it should not interfere with production runtime code.
• ✔️️️ you’ll need 2 essential configuration entries: the Pants configuration -how Pants runs Pylint- in pants.toml and the Pylint configuration -how Pylint runs-, in pyproject.toml (or .pylintrc)

Let’s walk through a working example! We manage Prefect flows and we agreed on a naming convention for our Prefect tasks: their definition name must end with _task . Fair enough, here’s an example implementation of a custom checker that would do the job:

If you’re not familiar with Pylint checkers, here’s what it does if I were to break it down into five steps:

• visit FunctionDef AST nodes on all linted modules. If you’re not familiar with AST, check Python’s AST lib and Astroid.
• for each function definition, check if it is decorated by a called decorator, e.g. @decorator(potential_args) , this excludes those of the form @decorator
• check if the decorator function is a prefect.Task
• if the definition name does not abide by the convention, add a Pylint message
• the register function tells Pylint to use that new checker

Let’s say you’ve added the project pylint_plugins in your repo, this piece of code could be located in pylint_plugins/tasks.py.

You’ve got your checker assembled in a dedicated project, here is how it should look in the configuration:

Now you’ll need to tell Pants it should look for 1st-party Pylint plugins in this project:

Key items in this configuration:

• ✔️️️ the resolve, that matches the one defined in the BUILD file
• ✔️️️ root patterns! As required per Pants in their documentation
• ✔️️️ Pylint source plugins: the source is located as follows {root_pattern}/:{target_name} . ⚠️ This tells Pants to read that location when running the tool Pylint, it does not tell Pylint to use it! This comes in the next configuration element

Now you can set up your Pylint configuration as simple as this:

How does this work? We’ve already told Pants to read the location pylint_plugins. Now Pylint, when run, has access to everything under this separate project, and we ask it to load the plugins located in tasks module (remember to use modules and not files).

It’s time to lint!

Let’s have a dummy task definition, that does NOT respect our convention:

We can now run ./pants lint :: and you should get a similar result:

You’re all set up!

From a convenient project to a production-ready tool:

• ⭐️ performance? the time cost of the added plugins? Even if that piece is called during the CI, you might be questioning the performances. In fact, unless you add hundreds of intricate AST plugins, the added plugins won’t cost a lot, or won’t cost anything extra. Also, you can easily cut the volume of code being linted by covering only the edited ones: ./pants --changed-since=origin/master lint (remember to cover all files the first time you run your plugin though!). If that’s not enough, Pylint can run with several jobs as well!
• ⭐️ the importance of keeping Pylint plugins in a separate project! As written above, it’s more convenient to have the plugins in a separate project for several reasons. First, that’s a CI tool, not a runtime piece of your software. Second, by putting more and more 3rd-party tools in your plugins, you might start interfering with your runtime 3rd-party libraries. In fact, we’ve reached a point where the versions of pylint and astroid conflicted with some other 3rd-party libraries we use. Moving to a separate project wiped the problem out.
• ️️️️️️️️️️️️⭐ ️linting does not solve bad architecture! A highly-linted code is not necessarily a good code. But it should make it readable according to industry standards.
• ️⭐ ️standard linting rules are relevant. Before jumping on custom checkers, try to apply all default Pylint rules.

Hope this will help you apply your conventions in an industry-fashioned way. Feel free to reach out for questions, suggestions, or examples of your best checkers!

A sample repository with all the code snippets above is available at https://github.com/DoctrineLegal/demo-pants-pylint