Python CLI Tricks That Don’t Require Any Code Whatsoever
Start webserver, open browser, parse JSON, compress files and much more with Python commandline tools and without opening a single file or writing any code
Out-of-the-box, Python standard library ships with many great libraries some of which provide commandline interface (CLI), allowing us to do many cool things directly from terminal without needing to even open a .py file.
This includes things like starting a webserver, opening a browser, parsing JSON files, benchmarking programs and many more, all of which we will explore in this article.
Benchmarking
The one Python module you likely already saw being used directly from commandline is timeit. It can be used for simple benchmarking:
In many cases, single line of code is not enough to perform a test — if you need to do some setup, then you can use -s option to - for example - create variables or import modules. This option can also be used multiple times.
As you can see from the 2 examples above, the number of loops can vary widely — in the first example we have 50,000,000 loops, while in the second one we have only 2000 loops — this is based on the time it takes to execute the code once. To force certain number of loops, you can use -n X option. The output also says "best of 5", meaning that it runs the code in 5 loops of 2000 executions each, you can change that with -r X option.
If you need a bit more powerful benchmarking tool, yet want the simplicity of timeit, then you can try pyperf:
pyperf has very similar interface to basic timeit, it even provides subcommand of the same name. This command will produce benchmarking stats ( result.json in the above example) which can be then inspected with stats subcommand. You can also create histogram using hist subcommand and the same file.
pyperf can be also used to make performance comparisons between different versions of Python - this can be handy when evaluating improvements in newer Python versions.
As you can see, pyperf has clear advantages over basic timeit, if you want more reasons why you should use pyperf, then see the comparison in pyperf docs.
Finally, you can also use builtin cProfile module, which provides tools for profiling of Python programs, simple invocation would look like:
This test was run against a sample code that computes e to the power of X. This kind of profiling can be useful, but is quite limited. cProfile only gives us information about function calls, not about individual lines of code - if you call some particular function such as append in different places, then it will be all aggregated into single line in cProfile output.
For a deep dive into Python profiling, see my previous article:
Running Servers
Let’s say you need a simple HTTP server for testing or want to serve files from a directory. Python’s http.server makes this super easy:
This is very convenient for testing, but it’s definitely not production-ready. If you need a proper server for your Python web application, consider using WSGI server such as Gunicorn.
Also, if you don’t mind writing a bit of code, then http.server can be customized a bit more, see docs for examples and allowed methods/options.
If — instead of just serving files — you need full FTP server with upload, download, authentication and more, then pyftpdlib will solve that for you:
Here we use pyftpdlib to start a server listening on port 2121, serving /tmp/ directory with user someuser and password somepass. We are then able to connect to it using ftp Unix command.
pyftpdlib should be sufficient in most cases, but if you're looking for alternatives, ten make sure to check out FTP server provided by framework. You can install the library using pip install Twisted and run the server with twistd ftp ... .
Debugging
While debugging from your IDE of choice is probably more convenient, Python provides tools for debugging directly in terminal. One such tool/module is pdb.
There are multiple ways to use the pdb module, first one being through python -i ... option:
Here we tried running simple script that does return 0 / 0, which obviously throws ZeroDivisionError. Thanks to the -i option, after the program exits we go into interactive REPL session where we can debug the program. We do that by importing pdb and starting post-mortem debugger with pdb.pm(). From there we can use pdb commands to inspect the environment and to find out why our program crashed. In this example we use l(ist) command which shows surrounding lines.
Another option is to execute the program with pdb from the beginning:
This will break the execution on the first line, from there you can use pdb commands to set breakpoints (b), step over (n), etc. We also used the -c option to set a breakpoint on line 4, this is denoted by "B" after the line number.
pdb is great, but it's not the only debugging tool Python provides - if you are invoking a Python script from a commandline and it's giving you very non-descriptive error, such as segfault you can try the following:
Executing import ctypes; ctypes.string_at(0) thrown segmentation fault without any actual context. To get more info we use faulthandler module which dumps traceback.
Finally, not just for debugging, but for general inspection of program execution, you can also use trace module:
In this example we use the sample program from the earlier profiling example, this time invoking it with the trace module, supplying --count option to count number of executions of each line, -C for directory name ( cover) where we want the reports to be stored, as well as --summary option to give us quick summary in stdout.
In the snippet, you can also see contents of cover/some-code.cover, which shows how many times was each line in the file executed.
The trace module provides couple more CLI option, for complete list see docs.
Parsing JSON
You surely know that Python standard library includes json module, but did you know that it also provides commandline interface?
This way you can validate and pretty-print JSON on commandline. This is especially useful if you’re working on system that doesn’t have more JSON processing tools, such as jq.
Additionally, if you provide --json-lines option, you can also parse jsonl format - that is - a file containing multiple JSON objects.
Compression and Packaging
Python’s standard library also includes a set modules for compressing/decompressing archives, such as zlib, gzip, bz2 or tarfile. Some of these also provide commandline interface:
In this example we have 2 files called words1.txt and words2.txt. We use gzip module to compress these individual files. Disadvantage of gzip is, that it's able to compress only individual files and not directories. If we want to create an archive from a directory we have to use other modules:
Here we use zipfile module to create, inspect, and extract files from archive. You can also use tarfile instead, which provides identical interface.
Also related to archiving — Python provides zipapp module which can create ZIP files of Python code, which can be then executed by interpreter:
Here we have directory named app containing hello.py file with say_hello function. To create executable archive from this directory, we use python -m zipapp, specifying the directory, interpreter path (-p), which also makes the file executable, and entrypoint (second -m), which is the name of file and the function. This outputs app.pyz which can be directly executed.
This module can be used as a simple way of distributing Python application to users. All that user needs is compatible Python interpreter.
Beyond this simple example, it’s also possible to create a standalone application with its dependencies packaged — for a complete walk-through of the module see tutorial at RealPython.
Starting a Web Browser
We know that Python comes with “batteries included”, but even then, from time-to-time you discover modules or tools that are little unexpected in language’s standard library. One such module is — in my opinion — webbrowser:
This module allows you to open a browser window (-n) or a tab (-t) (with specified URL) directly from terminal or from your Python code.
The CLI will always open only default web browser, for non-default browser use:
Here, in google-chrome %s, the %s gets substituted with the URL, and gets invoked as if it was called in terminal. Therefore, the command - google-chrome in this case - has to be available in your shell.
Python-Powered Shell
Python comes with a lot of great tools, but there are many more tools out there. Great example of that is Xonsh — a Python-powered shell:
It combines the simplicity of Unix shell (bash), with the power of Python — allowing us to run both commands like ls or pwd shown above, as well as any Python code directly in terminal, which makes it somewhat similar to IPython, but suited more for shell scripting.
Closing Thoughts
Python is great at many things — including providing some CLI tools for common tasks. Almost all of these are part of Python’s standard library, which provides many more modules and libraries that can help you write better code or simplify common tasks. Finding all the useful stuff is really just matter of reading the docs, and I strongly recommend reading through the library index — I’m positive that you will find something useful that you didn’t know was there.
This article was originally posted at martinheinz.dev
Become a member and read every story on Medium. Your membership fee directly supports me and other writers you read. You’ll also get full access to every story on Medium.
