python.mk

As a backend software engineer I find myself solving a lot of problems with a quick python script. These scripts are mostly unsupported by the ergonomics that I take the time to set up in my long term projects. To solve this I created python.mk, a collection of my favorite tools for python scripting conveniently bundled into a Makefile.

Python-mk has strong opinions about process and development tools for python but doesn’t care about your text editor. It is extremely portable — you don’t even have to have python installed. Its only dependencies are docker and make.

Features

When starting a new project, copy the python.mk into the directory and run make -f python.mk install. It creates boilerplate files and directories and defines several common tasks to help develop and run the script. The default project structure looks like this:

? tree -a
 .
 ├── Dockerfile
 ├── .gitignore
 ├── main.py
 ├── Makefile
 ├── modd.conf
 ├── python.mk
 ├── requirements.txt
 ├── scripts
 │ └── modd
 └── test_main.py

Docker is used to create the environment where the script is run. Run make build and a slim python base image is pulled and all the dependencies are installed. Customize the Dockerfile and extend your requirements.txt to ensure all the desired dependencies are available. It obviates the need for virtual-env, or managing python versions.

Modd is a binary that can watch your source files and run commands automatically when they are changed. It is configurable with a modd.conf file and is run using the make watch command. By default it will run make build when the Dockerfile or requirements.txt is modified and run make test if any python file is modified. You can customize this configuration to react and run any commands you prefer.

There are three different parts to the make testcommand.

1. Code formatting with black which can also be independently run with make format.
2. Static type checking with mypy which can also be independently run with make check.
3. Finally pytest is called and uses automatic discovery to find and run all functions that start with test_*.

To run the script use the make run command. As with watch, format, check and test; run is a proxy for the docker run command. Docker is configured to open a volume to the project directory and the user id and group id are passed so that any files that are generated by the script have the same permissions as the host user.

The final command is make console which also uses docker run but adds interaction and tty support. It drops you into a bash shell in the working directory. This command is useful while debugging the environment or is a way to run infrequent commands that aren’t worth writing a custom make command for.

Extending

Three environment variables are sourced by python.mk and it is recommended that they are customized and set during the initial installation.

• APP_NAME is used as the script filename. The default is main.
• IMAGE_TAG is the name and version of the docker image that gets built. The default is pythonmk:latest. It is really important that this is overridden otherwise multiple projects would use the same docker namespace and this would cause excessive build times.
• MAINTAINER is just the email of the user or team, it is set in the Dockerfile. The default is person@example.com.

A simple way to specify all these variables during the initial installation is to run a command like:

? APP_NAME=myapp \
  IMAGE_TAG=myapp:latest \
  MAINTAINER=me@c11z.com \
  make -f python.mk install

The install generates a parent Makefile, sets your custom variables and inherits the python.mk. This file is really useful for overriding and extending functionality. For example I have another project (or-the-whale) that is a NLP analysis of Moby Dick. It has spaCy as a dependency, which isn’t yet supported by mypy. The make check command is overridden to add the flag --ignore-missing-imports in the parent Makefile.

include python.mk
 …
check: format
    @docker run \
    --rm \
    --user $(UGID) \
    --volume $(CURDIR):/script \
    $(IMAGE_TAG) \
    python3 -m mypy — ignore-missing-imports /script
 …

Also in this project the main python file takes command line arguments to run different experiments. I created a new make command — make call — which uses an environment variable ARGS defined in the parent Makefile like this:

ARGS?=”noop”
 …
call: build_quiet
    @docker run \
    --rm \
    --user $(UGID) \
    --volume $(CURDIR):/script \
    $(IMAGE_TAG) \
    python3 /script/$(APP_NAME).py $(ARGS)
 …

Now any experiment can be run by:

ARGS=”experiment_1" make call

Conclusion

Like any other bootstrap system, python.mk makes a lot of opinions about what a good development environment is. Personally I love code formatters, static type checking and I feel really comfortable with make. If you feel like python.mk is missing something or can be improved I would love chat about it in the github issues or ping me on twitter @corydominguez.

Inspired by elm.mk which itself was inspired by erlang.mk.