python.mk

c11z
c11z
Oct 28, 2018 · 4 min read
Image for post
Image for post

As a backend software engineer I find myself solving a lot of problems with a quick python script. These scripts are mostly unsupported by the ergonomics that I take the time to set up in my long term projects. To solve this I created python.mk, a collection of my favorite tools for python scripting conveniently bundled into a Makefile.

Python-mk has strong opinions about process and development tools for python but doesn’t care about your text editor. It is extremely portable — you don’t even have to have python installed. Its only dependencies are docker and make.

Features

? tree -a
.
├── Dockerfile
├── .gitignore
├── main.py
├── Makefile
├── modd.conf
├── python.mk
├── requirements.txt
├── scripts
│ └── modd
└── test_main.py

Docker is used to create the environment where the script is run. Run make build and a slim python base image is pulled and all the dependencies are installed. Customize the Dockerfile and extend your requirements.txt to ensure all the desired dependencies are available. It obviates the need for virtual-env, or managing python versions.

Modd is a binary that can watch your source files and run commands automatically when they are changed. It is configurable with a modd.conf file and is run using the make watch command. By default it will run make build when the Dockerfile or requirements.txt is modified and run make test if any python file is modified. You can customize this configuration to react and run any commands you prefer.

There are three different parts to the make testcommand.

  1. Code formatting with black which can also be independently run with make format.
  2. Static type checking with mypy which can also be independently run with make check.
  3. Finally pytest is called and uses automatic discovery to find and run all functions that start with test_*.

To run the script use the make run command. As with watch, format, check and test; run is a proxy for the docker run command. Docker is configured to open a volume to the project directory and the user id and group id are passed so that any files that are generated by the script have the same permissions as the host user.

The final command is make console which also uses docker run but adds interaction and tty support. It drops you into a bash shell in the working directory. This command is useful while debugging the environment or is a way to run infrequent commands that aren’t worth writing a custom make command for.

Extending

  • APP_NAME is used as the script filename. The default is main.
  • IMAGE_TAG is the name and version of the docker image that gets built. The default is pythonmk:latest. It is really important that this is overridden otherwise multiple projects would use the same docker namespace and this would cause excessive build times.
  • MAINTAINER is just the email of the user or team, it is set in the Dockerfile. The default is person@example.com.

A simple way to specify all these variables during the initial installation is to run a command like:

? APP_NAME=myapp \
IMAGE_TAG=myapp:latest \
MAINTAINER=me@c11z.com \
make -f python.mk install

The install generates a parent Makefile, sets your custom variables and inherits the python.mk. This file is really useful for overriding and extending functionality. For example I have another project (or-the-whale) that is a NLP analysis of Moby Dick. It has spaCy as a dependency, which isn’t yet supported by mypy. The make check command is overridden to add the flag --ignore-missing-imports in the parent Makefile.

include python.mk

check: format
@docker run \
--rm \
--user $(UGID) \
--volume $(CURDIR):/script \
$(IMAGE_TAG) \
python3 -m mypy — ignore-missing-imports /script

Also in this project the main python file takes command line arguments to run different experiments. I created a new make command — make call — which uses an environment variable ARGS defined in the parent Makefile like this:

ARGS?=”noop”

call: build_quiet
@docker run \
--rm \
--user $(UGID) \
--volume $(CURDIR):/script \
$(IMAGE_TAG) \
python3 /script/$(APP_NAME).py $(ARGS)

Now any experiment can be run by:

ARGS=”experiment_1" make call

Conclusion

Inspired by elm.mk which itself was inspired by erlang.mk.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store