Code Design Principles for Public APIs of Modules

Published in

code.kiwi.com

10 min readOct 13, 2017

The following article is from Kiwi.com’s internal engineering handbook. We thought it could come in handy for others as well, so we decided to share it with the world here.

This document should leave you with some idea on how to approach problems where you have to create a new module from scratch, or come up with a way to completely redesign an existing one.

Have nothing left to take away

Perfection is achieved not when there’s nothing left to add, but when there is nothing left to take away

— Antoine de Saint-Exupéry

Let’s examine the example of kw.booking.additional_booking_management (an internal module for managing addons for flight bookings.) This code can't be found in master anymore (obviously, we're not going to just leave it lying around to be a bad example) but that just means we've seen how adhering to these rules works in practice.

This module name describes exactly what it’s for: managing additional bookings. But is there anything we can take away here? Carefully thinking about it, we realize that the suffix _management actually adds no value. It's just a generic word we can add to anything, while it conveys no useful information. If we just call it additional_bookings, what could you assume it does with ABs other than managing them? There's not a lot else you can possibly do.

Why are we wasting so much time on thinking?

So we got rid of one third of the name, saving us space in the code — valuable space that we need to convey more complex thoughts.

If you don’t see how this matters, just compare these two lines:

total_building_area_in_square_meters = number_of_buildings *(width_of_buildings_in_cm / 100) * (length_of_buildings_in_cm / 100)total_area_m2 = len(buildings) * (width_cm / 100) * (height_cm / 100)

You don’t lose any meaningful information, but it takes about half as much space in the second example. Using half as much text means way faster scanning of the code for the human reader — if you squint your eyes so the letters themselves are illegible, you can still sort of make out which part of the code it is, just by seeing something like •••••• = len(••••••) * (•••••• / 100) * (•••••• / 100). It's already useful to just see some sort of a calculation is done on this line, not something like if •••••• > ••••••: or def ••••••(••••••, ••••••):. In the longer version, you just wouldn't be able to see the entire line of code with one glance. This still sounds like it's not a huge deal, but now you should consider the context in which you're most often reading code. Some examples would include when you’re trying to:

Track down how a variable could get a specific value in a production error: walking through like 10 levels in the stack, under lots of PagerDuty and Slack panic pressure
Understand how exactly you should call this module in the code you’re writing, without needing to understand its inner workings
Find an example of how this module uses logging, databases, or anything else, so you can copy-paste something to your own code
Track down where exactly you should call the new function you’re about to write for some brand new feature
Make sure this code doesn’t break the surrounding changes you’re reviewing
Make the same kind of change repeatedly in all places in the repo (refactoring DB usage, concurrency, or fixing static analysis errors)

There’s something that all these examples have in common. They all keep your brain overloaded, deep in conscious, focused thought, with your short-term or working memory full of the information you need to keep in mind to complete your original goal. Ideally, you’re already in the zone, your thoughts flow without any doubts or impediments, and the only thing holding you back is that piece of information you’re searching for. Things can go one of two ways here. In the better scenario, you would scan through the code, without actually reading or understanding it, to find the one line you really need and then move on with your task.

But what can also happen, is that the code proves to be too much to just thoughtlessly scan. Lengthy lines, roundabout ways of manipulating values, and excessive use of variables can all lead to changing your ‘find what I need here’ task, to depend on a new ‘understand this code’ task. While previously you were able to rely on your instincts and experience to understand just enough of the code to make progress, now you need to switch to make a conscious attempt to understand what is happening, since your working memory is now storing new items. If the code is overcomplicated, one such extra item could be ‘buildings_count is just len(buildings)'. This has no relevance to your task if you’re just trying to see how the building count is passed or bound to the logging system, but still wastes valuable space in your short-term memory.

So now we understand that consequences can be way more severe than just ‘making people read for one second longer.’ You knocked an engineer out of their flow and interrupted their original task, which they might not be able to resume without spending several minutes on getting all their contextual information back. Now we need to reach another realization: not only is this more severe than it seems, but it’s also way more frequent. This one doesn’t require much explanation from my side, just do the following without thinking: come up with the number of times you expect people to read the last piece of code you wrote in the next year. Now give it a bit more thought, and come up with accurate estimations for:

the number of engineers who often work on this module
the probability they will need to look at it any day
the number of engineers who don’t work on this module, even from completely different teams
the probability they will look at it anyway, to solve issues, to understand how it fits in with their code, because they’re refactoring something your code uses, or even just out of curiosity
the number of days this code will stay in the codebase — if you’re thinking ‘we’ll refactor it soon anyway’, stop kidding yourself and come up with a real estimate

Now put these numbers together, and be amazed and how much more it is than you originally thought.

Sidenote: Don’t optimize only for scannability

While scannability is very important, it’s not the only aspect of readability. Compare this to the above:

a_m2 = len(bdgs) * (w / 100) * (h / 100)

While it looks even more like an equation, and it might be even easier to scan if you’re squinting and not trying to actually read the code, it took a huge hit in understandability, which you still require for working on this specific piece of code. I guess my message here is:

Readability = Scannability + Understandability
People are naturally biased to favor Understandability, even though Scannability is just as important, so you need to actively fight this bias.

It can be okay to make people learn new things

So now we’re back to kw.booking.additional_bookings. Something still isn't right here. This name, additional bookings… It's pretty long, it can't be the simplest way there is to describe passenger service requests and other additions… attachments… ancillaries… extras. An extra. Huh. Again, it's not something that has a perfectly fitting dictionary definition, but if you imagine seeing a booking, and see it described as having '3 extras,' wouldn't you be able to intuitively understand the general idea behind it? And won't you just simply recall that the next thousand times you see the phrase?

We have an easy way to propagate the information to people — the name is self-explanatory if you just think about it for a bit
The phrase or concept is popular enough for natural selection to come into effect — when there’s enough usage over time, the better name will catch on and drown out the old name(s).
It’s easy to keep this change consistent, there’s no need to worry about having minor variations show up. This would be a problem for instance if we tried to do something like adding standard creation timestamp columns to all tables in the database. If we decide that created_at is the best name for that, but don't actively enforce it, it's only a matter of time until someone names their column created instead. Or inserted_at. Or creation. Or creation_date. In the case of created_at, since it's so easy to inadvertently create variants, we had to create a repo for declarative database models and commit the name in a reusable piece of code.

Everything is perfect to make this into a thing people should just learn! Since the above three points all work in our favor, the short term pain of making people re-learn this is worth it for the long-term benefits.

Think about the user first

We’re all humans, and it’s very easy to lose yourself in focusing on just completing your task. But the truth is, using your own code internally is not where the real problems lie. What’s way more important is how other people will see your module, when they don’t care about the internals. Not only will this happen more frequently than you using your module from inside your module, but more importantly, a badly designed module harms the outside users way more than it can possibly harm you.

If you care about tending to your code and keeping it clean, so that your work is easy, you can end up with a nice, clean design; for instance placing your data-mangling functions like get_additional_detail in models.py. This makes it easy for people to work on kw.booking.extras, great! But is it easy to work with kw.booking.extras then? Sadly, it isn't. If someone just wants to use your extras module, now they have to go hunting to figure out how you designed its internals and how you can actually use the code. Turns out you'd have to do from kw.booking.extras.models import get_additional_detail, which is just unpredictable; no one would guess this to be the usage without seeing the code.

How would we solve that? Easy, just act as if you were a user and think of how you’d expect to use the module, before you start working on it. There’s actually a very straightforward way to accomplish this: write tests. Tests are basically just you using your own code.

We’d arrive at the following conclusions:

Accessing kw.booking.extras.models is just unnecessarily inconvenient. A simple import in your __init__.py can let your users write the import in a simpler, more obvious way: from kw.booking.extras import get_additional_detail
If the module author thought of the users, and exposed everything you need in __init__.py, we could change our imports to make the function calls more clear with the power of namespacing: from kw.booking import extras and use it like extras.get_additional_detail(123, 456).
get_additional_detail is a silly name for our function which returns an extra from the database by ID in the form of a dictionary. It'd make more sense as get_extra_details, so now we have from kw.booking.extras import get_extra_details.
But if the intended usage is extras.get_extra_details(123, 456), why are we mentioning the word extra twice? It makes just as much sense as extras.get_details(123, 456), or if you want to be extreme, even extras.get(123, 456).

And this was the story of how we went from

from kw.booking.additional_booking_management.models import (
    get_additional_detail, add_additional_flights, expire_additional_bookings
)additional_booking = get_additional_detail(bid, abid)
add_additional_flights(additional_booking)
expire_additional_bookings(bid)

from kw.booking import extrasextra = extras.get(bid, eid)
extras.add_flights(extra)
extras.mark_expiries(bid)

Function signatures are just as important

So far we’ve been focusing mostly on correctly naming, and correctly placing things. There’s one more thing that users constantly encounter during usage: the actual usage.

When people try to use a function or method you’ve written, they ideally don’t need to delve into more detail than just checking the list of parameters. Consider for example requests.get. You'll see parameters like url, headers, timeout, auth. We can learn some lessons from Kenneth Reitz's library.

Keep the number of required parameters to a minimum (this will usually be just one or two per function)
Set sensible defaults for the optional parameters, which would make it easy to work with the function in an interactive interpreter. This is not the goal, but it’s the easiest way to think about what the default should be — what would you want the values to be if you were just testing the function right now?
Prefer to consolidate parameters when it makes sense: instead of asking for bid, booked_at, partner, market, ask for just booking, and access booking.id, booking.timestamp, booking.partner, booking.market
Order the optional keyword arguments by their expected frequency of use.
Explicitly disallow confusing usage of function calls. If you have a function like book_flight(id, sandbox=True), people will inevitably start using it like book_flight(123, False), where nobody would be able to understand what the second parameter means just by looking at it. You can define your function like this instead: book_flight(id, *, sandbox=True), and people will be forced to write out sandbox=False every time they use your function. This feature is available only in Python 3, however.

The direct object

Think about what the ‘direct object’ of your function is, if we can use a grammar analogy for a minute here. Let’s say you have this sentence: ‘I went to buy honey from Tesco at 10pm.’ Here, the verb to buy requires a direct object, which would be honey. The verb makes no sense without a target—you can’t just say ‘I went to buy,’ so we always need to specify this. We also specify two more things: the source of the honey and the time of the purchase, but these are optional details without which the sentence wouldn’t become meaningless.

So how does this relate to programming? The grammatical structure here actually maps pretty well to function signatures. If we had to write this sentence down as a function call, it’d be something like buy('honey', source='Tesco', time=arrow.get('22:00')). Notice how mapping the buy action to code still leaves it with the same sort of semantic restrictions. A buy call still makes no sense just like this, without a target: buy() — just like a requests.get() call makes no sense without a URL.

Everything else, however, can be removed, and that’s why we make those parameters optional keyword arguments, with sensible defaults.

In summary

If I had 5 seconds to tell you something new, it’d be a plea to think about scannability and consider what your users will see. For all the rest, please just scroll up and bear through the raw writing style — it’s an engineering handbook, after all!