Speed up your Python development workflow in 5 minutes with pre-commit and Makefile
Introduction
When developing Python code we are constantly adding and committing changes. However, nothing stops us from committing low quality code, e.g. code containing unused imports, unformatted code or functions that do not work correctly. Therefore, to make sure the code we commit is of sufficient quality, we use formatting-, linting- and testing tools to check our code. These checks will result in increased code readability, maintainability and quality and will improve collaboration amongst colleagues. We can run these checks manually, but this is time consuming and prone to errors. Therefore, we recommend automating these checks to simplify your development workflow.
In this blog, we are looking into a tool stack that helps us running these checks. We will highlight such a setup with the tools pre-commit and Makefile. When set up right, they complement each other. We will describe why you would want to use them, how they work and what a minimal setup looks like for your new project.
Format, lint, test!
To ensure our code quality we often make use of formatting-, linting- and testing tools. Each of the three has its own purpose and complement another:
- Formatting tool automatically reformats entire files to a single style. It makes code reviews faster and improves collaboration.
- Linting tool performs static code analysis. It checks for code errors, code with potentially unintended results and dangerous code patterns.
- Testing tool that can be leveraged to run tests against your code, to verify functionality.
In our setup we use Black as formatting-, Pylint as linting- and Pytest as testing tool. Black is a PEP8 compliant formatter without much to configure. Pylint comes with a code rating and is the most used linting tool for Python. Pytest makes it easy to write unit tests without boilerplate.
Automatically run checks with every commit
Pre-commit is a tool that is used to run scripts (hooks) to automatically identify issues in your code. Configured hooks are triggered when committing your code to Git.
To use pre-commit you need to:
- Install pre-commit with
pip install pre-commit - Create a pre-commit configuration file
.pre-commit-config.yaml. This file will contain the hooks you want to run and their settings. - Install git hooks with
pre-commit install
There are many hooks which can be used to check your code, like hooks from pre-commit itself, or third party tools, like Black and Pylint. A minimal example of a .pre-commit-config.yaml file is shown below:
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v3.4.0
hooks:
- id: check-docstring-first
- id: check-merge-conflict
- id: trailing-whitespace
- id: end-of-file-fixer
- id: check-yaml
- id: check-ast
- repo: https://github.com/ambv/black
rev: 20.8b1
hooks:
- id: black
- repo: local
hooks:
- id: pylint
name: pylint
entry: pylint
language: python
types: [python]This setup ensures that for every commit, pre-commit will automatically run the set of specified hooks. First a set of hooks from pre-commit reformats and checks your code. Secondly, Black reformats your Python code. Finally, Pylint scans your code and gives an overall rating. Only if your committed code passes all checks your commit will be completed. Otherwise you (or the hooks) have to change the code accordingly and commit again.
Note that Pytest is not included in our pre-commit configuration as running tests is usually expensive. A list of useful pre-commit hooks can be found here.
Easily run checks with Make shortcuts
Using pre-commit, checks are automatically ran on each commit. However, we might also want to run checks before committing code. Although running checks can be done with separate commands, a more efficient setup can be achieved by using a Makefile. A Makefile makes it possible to create shortcuts that easily run a task (such as your pre-commit) or a set of tasks (such as pre-commit and testing). These shortcuts can then be triggered with the make command in your terminal. An example of a Makefile is shown below:
## Run all pre-commit hooks
.PHONY: precommit
precommit:
pre-commit run --all## Lint your code using pylint
.PHONY: lint
lint:
python -m pylint --version
python -m pylint src## Run tests using pytest
.PHONY: test
test:
python -m pytest --version
python -m pytest tests## Format your code using black
.PHONY: black
black:
python -m black --version
python -m black .## Run ci part
.PHONY: ci
ci: precommit lint test
With this setup you can run multiple tasks with simple shortcuts (stated after .PHONY and opening each task). The make precommit command runs all your pre-commit hooks on all your Python files. The make black command runs Black on all Python files in your current working directory. The make lint command runs Pylint on your src directory. The make test command runs Pytest using the tests directory. Finally, the make ci command runs previously described tasks in sequence.
Conclusion
Pre-commit and Makefile are great tools to deliver quality Python code in an early stage. They will speed up your development workflow which results in more time for code logic. Both have their own purpose:
- Pre-commit assures that your committed code is of expected quality.
- Makefile simplifies running local quality checks while developing.
Next steps
- Implement pre-commit and Makefile locally, to automatically run your checks.
- If you have a CI pipeline set up, leverage Makefiles to make it easier to run the checks from your CI pipeline locally. Now, instead of pushing and waiting for feedback of your CI pipeline, you get feedback directly.
- To ensure not just you, but all your colleagues commit quality code, add your pre-commit configuration to your CI pipeline too. This way everyone will be held to the same standards.
As it is easy to integrate pre-commit and Makefile, make sure you try them in your new project! If you need help, please respond in the comments below.
