Python Template

I come from the world of java. In java everything is organized and there is a structured way to do it. Whether you use maven or gradle, your projects are very structured, and any person that looks at your code knows how to navigate the project.

Python is a new world and has a lot of different standards and there is no single way of structuring your project and what tools you need to use.

Before we drill down into the python options, I would like to cover the basics that I feel should be in any project of any language.

Testing

No matter which language it is, you want to have unit tests (in case you don’t do it the-importance-of-unit-testing). You might even add integration tests. So what needs to be done for testing no matter what the language?

You want to separate your production code from testing code. You don’t want to deploy test code as part of your production code, so the testing code needs to be separated from the production code (different folders).

Run tests from CLI. Since you will want all your tests to be run as part of the CI, you need to be able to run all tests from the CLI (even if for debugging it is easier from the IDE).

Code Coverage

Code Coverage. Tests are great, but you need the ability to measure how much of your code has been covered by the tests. You should usually aim for 70–80% of your code to be covered by tests.

Coding Standards. Each language has its own nuances, but there needs to be some standard as how to write the code (spaces, indentation, comparability, etc… )

Building

Every project needs a standard way to build the project. Building the project should be the same in the CI as in the local dev environment.

Every projects has dependencies, so you need a standard way to register all dependencies and an option to download all dependencies. Also you need a way to have product dependencies and testing or dev dependencies. You don’t want dependencies needed for dev to be part of the production asset.

Python Tools

I will bring here my personal preference to the magic sauce for every python project that I write.

Building

In the past, most projects used requirements files to handle dependencies, with at least one for production and one for development. It is time to move on to more modern tools. I use poetry for this. Poetry can handle both types of dependencies, it knows how to check for updates, and even knows how to deploy a python library to your repository without the need to create a setup.py file.

Testing

Now that we have a basic project that can be deployed, we want to add tests.

We will need to create all tests in a test folder so that poetry does not package it as part of the project. Personally I prefer the pytest framework. There are many other frameworks out there, the following blog: top-python-testing-frameworks has a short summary of them.

I also usually have two folders: tests, integration-tests (if you are externalizing a server you might want an api-test folder). All types of tests are written with pytest, but the context of each is different, and the time to run each can also differ.

Of course you can add settings for pytest in a pytest.ini file, for example:

[pytest]log_cli = 1log_cli_level = INFOlog_cli_format = %(asctime)s [%(levelname)8s] %(message)s (%(filename)s:%(lineno)s)log_cli_date_format=%Y-%m-%d %H:%M:%S

But since you are using poetry you need to add these entries in your pyproject.toml

[tool.pytest.ini_options]log_yoda = truelog_yoda_level = "INFO"log_yoda_format = "%(asctime)s [%(levelname)8s] %(message)s (%(filename)s:%(lineno)s)"log_yoda_date_format = "%Y-%m-%d %H:%M:%S"

CI — Testing

When coming to run the tests, you need to install the proper version of python and then run the tests. For this we use tox. Tox will install python (you can setup more than one version), and then run all pytests in your project. This way you do not need scripts to install python, and you can check you application against multiple versions of python.

An example of a tox.ini:

[tox]skipsdist = trueenvlist = py39[testenv:tests]whitelist_externals = poetrycommands =poetry install -vpoetry run pytest {posargs} tests — cov-report term — cov=yodacoverage html — fail-under=70setenv =[testenv:integration]
[coverage:report]
skip_empty = true

Code coverage

In addition we have setup code coverage for running our unit tests. You can set your own limit as to what percentage you want to fail the build. This way we can validate that our tests are covering the needed amount of code.

Code Styling

Last but not least, there are a lot of standards for python coding. The basic formatting of code is based on pep8. But since pep8, there is a new and better formatter called black. Black is based on pep8, but will not only validate your code to be according to the standard, black will fix your code.

One of the very annoying issues, is how to write python imports. There are different ways, and each IDE, reformats the file according to it’s own way. This leads to changes in git that are not real changes. To solve this we used a library called isort. isort has its algorithm for how to sort imports, and will always give you the same order for a given set. This way no matter which IDE you use, isort will reorganise the imports accordingly.

Code Enforcement

Now that we have defined black as our code formatter, and isort for our imports, we want to enforce the usage of them. We do not want to hope that all our programers have installed the IDE plugin. The solution for this is git pre-commit. With pre-commit plugin, we can run all sorts of code as part of the commit process. If the pre-commit fails — there will be no commit. So of-course we will add black and isort to the pre-commit.

But once we have added pre-commit, we now have the option to do all sorts of other validations, like:

  • Check yaml files validity.
  • Check there are not big files being committed (by mistake).
  • Make sure no merge markers are left in code.
  • Check for private keys in code.
  • Make sure no debug statements (breakpoint()) are in code.

Summary

Python is a great language, and you can easily spin up a new project. But to do it properly requires skill and discipline. You want to use state of the art tools to manage and build your projects in a repeatable and reliable manner.

For a skeleton project see:

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Chaim Turkel

Chaim Turkel

Group Leader and Mentor @ Tikal Knowldege