Python PEP Retrospective
This is the story of how I ended up becoming the author of four Python PEPs, all of which were successfully approved and incorporated into the Python language specification.
The letters “PEP” stand for “Python Enhancement Proposal”. This was part of a process whereby people could submit ideas for improvements to the Python language and standard libraries, and those approvals would be considered by the Python maintainers.
I had met Guido van Rossum a couple of times in the early 2000s at conferences and users groups. Guido and I both shared an interest in compilers and language design; Guido of course created Python and had worked on several other languages that preceded it, whereas I had created a number of languages for interactive games, including SAGA (Scripts for Animated Graphic Adventures).
I had also implemented a number of DSLs (Domain Specific Languages) using Python as a foundation, one of which was used in my work at Electronic Arts to generate Visual Studio project files. As I once told Guido, “the only thing remarkable about the languages that I have created is the number of them.”
Because we both were language design enthusiasts, we had a lot to talk about.
I became fairly active on the Python developer mailing lists starting in 2005 or so. It was during this period that Guido announced his intent to create a new version of Python, “Python 3000”, which (unlike all previous versions of Python) would not be fully backwards-compatible with older versions.
Guido’s goal was to revisit several early decisions in the design of the language which had bothered him for years. Although the Python language had improved greatly in the years since its inception, certain unfortunate aspects of the language could not be changed without breaking every Python program in existence. Python version 3 was going to put those issues to rest once and for all.
Despite the playful title of “Benevolent Dictator for Life” or “BDFL”, Guido always made sure that the Python developer community was involved in decisions about the evolution of Python. So a new series of PEPs, starting with PEP 3000, was established to define what changes would made to the language, and developers were invited to submit suggestions for improvements. I was one of the first people to submit a new proposal.
A new mailing list was also created for discussing the design of Python 3. However, when creating the new mailing list, instead of typing “py3k” as he had intended, Guido accidentally transposed the middle two characters, so the actual name of the mailing list became “p3yk”. Rather than going back and fixing the typo, it was decided that this would be the official name of the project. (Humor has always been an important influence within the Python community, especially considering that the language was named after a British comedy troupe.)
In the following sections, I’ll discuss each of the four PEPs that I submitted during this period.
Two caveats: first, what I am about to describe is what I remember, but my memory may not be completely accurate. Second, not every idea that I floated on the mailing list made it to the PEP stage — some of my suggestions, in retrospect, weren’t all that great.
PEP 3101 — Advanced String Formatting
The motivation for writing PEP 3101 was that there was a lot of controversy on the mailing list how string formatting should work. A lot of people were unhappy with the existing “%” operator, but there wasn’t a clear consensus on what should replace it. There were a lot of suggestions made, but none of them had managed to garner much support.
My strategy in writing the PEP was to invent as little as possible. I gathered ideas from a lot of different sources, both on the mailing lists and off. Much of the design was inspired by the format strings from C# (I had done a lot of C# programming at EA). A lot of folks made suggestions which I adopted. My goal was to put together a proposal that the majority of developers would support, which eventually it did.
I’m not going to go over the details here, you can read them in the PEP. The gist of it is that in the old syntax, you would format strings like this:
print("Hello, %s %s" % (first_name, last_name))And in the new system, they looked more like this:
print("Hello, {0} {1}".format(first_name, last_name))It might not be clear at first glance why the second form is better, but the main difference is that it uses the standard Python function call syntax, which allows for great flexibility in mixing positional and keyword arguments in a single call.
A later PEP (not written by me) simplified this further to:
print("Hello, {} {}".format(first_name, last_name))After the PEP was adopted, I spent a lot of time writing the documentation for the new string formatting system, which was eventually incorporated into the Python language manual. The first implementation of the new system was done at a ‘hack-a-thon’ at Google which Guido had helped organize; most of the actual programming was done by Eric V. Smith, although I assisted with code reviews.
Looking back, I think that the design was rather over-engineered. There are tons of formatting options and pluggable hooks which a typical programmer would never use. If I were doing this today, I would probably create something much more basic and relegate the advanced features to an installable third-party package. For example, rather than adding format codes for customizing how numeric signs were printed, I would simply make that a separate call on the input data.
There are a number of factors which I think led to the design being as complex as it was:
- From a political standpoint, I think many developers would not have been happy with a design that wasn’t a superset of the previous one, that is, it had to be able to do everything the old system did. And the old system was inspired by the C standard library function ‘printf’, which had a lot of options.
- In that era (2006), it was still fairly common to have programs that generated reports in ASCII characters where the columns had to line up. Although that still happens occasionally today, most reports are generated in HTML or PDFs where the column widths are specified in pixels, not characters. It’s possible to do basic column layouts in ASCII with just a few simple formatting options, anything more than that is probably an edge case which is probably better handled by a specialized report-generating library.
In 2015, the PEP 3101 format strings were superseded by a newer, better formatting system called f-strings (defined in PEP 498 written by the same Eric V. Smith). It looks like this:
print(f"Hello, {first_name} {last_name}.")This new system still supports all of the format codes and hook functions as PEP 3101, but the syntax is much simpler and more intuitive. This is the method I would recommend for anyone learning Python today.
PEP 3102 — Keyword-only arguments
When you call a function in Python, you can pass arguments either by position or by keyword:
expect(items, 3) # positionalexpect(expected=items, actual=3) # keyword
You can also mix the two:
expect(items, actual=3)The only constraint is that every parameter must be assigned a value, either one that is explicitly passed in, or a default value specified in the function definition.
However, it’s a common idiom in Python to have arguments that represent ‘options’ or ‘flags’ to a function. These arguments are almost always specified using keywords. In fact, if there are a large number of parameters, then passing the values by position can lead to bugs, since most programmers find it hard to remember the exact ordering of all of the function’s parameters.
PEP 3102 defined a means whereby a parameter could be declared as “keyword only”, so that it could only be assigned a value using a keyword. Trying to pass that parameter positionally would generate an error.
The PEP also defined a way for a function to accept a variable number of positional arguments, while still accepting keywords. This would be useful for functions like “print”, which can take an arbitrary number of items to be printed, but still accept options:
print("My", "name", "is", "Fred", sep=":", file=sys.stderr)PEP 3115-Metaclasses in Python 3000
I’m not going to spend too much time on PEP 3115 since it is rather esoteric. Python has long had support for metaclasses, however the existing design had no way to preserve the order in which fields and methods were defined, since the metaclass only had access to a dictionary of class members, and standard dictionaries don’t remember the order in which items were inserted.
This is important if you are using Python classes to generate SQL table definitions or C++ data structures, where order is important.
The thing I remember most about this PEP is that it took a lot of lobbying to get it approved, since it took a while for people on the mailing lists to understand the problem it was trying to solve.
PEP 3119-Abstract Base Classes
This PEP I co-wrote with Guido. Actually, co-wrote isn’t quite accurate; all of the ideas in it were Guido’s, and I only wrote a few small sections. I mainly functioned as a technical writer because, although Guido had a clear idea of what he wanted, he (probably) felt that my explanations of the basic concepts were easier to understand.
The PEP is intended to make Python’s type system more robust by expanding the options for checking object types at runtime.
Afterward
I haven’t been active in the Python developer community for quite a few years. The main reason has to do with the nature of my work: although I do occasionally use Python from time to time, the majority of my programming is web-based and therefore in JavaScript (although in the last two years I’ve been writing almost entirely in TypeScript).
When I look back at my work from earlier years, I am often fascinated by which parts have stood the test of time and which have not. For any given decision that you made ten or twenty years ago, there are several possible reactions:
- “What idiot came up with that idea? Oh, right, that was me.”
- “Man, I spent so much time stressing about this detail, and the end none of it mattered.”
- “What amazing foresight! I actually thought of that?”
I imagine that the guys (and gals) who were around when COBOL was the new hotness probably thought it was the sweetest technology they had ever seen. But it is the fate of most technologies that twenty or thirty years later they seem clunky and sad. Python has fared far better than most of its cohort.
On the other hand — my involvement with the Python 3 language design is something that still looks good on a résumé :)
