Sometimes underscore isn’t so simple

Technical blog posts are often about issues or problems that are intricate, challenging, and/or visionary.

This post is about a problem that’s simple, easy to fix, and worthy of slapping your forehead. It’s a story about a bug happening at the intersection of two fine technologies.

The lowly underscore

In Python, the underscore (“_”) has multiple uses. The one that interests us today is when it represents an unused value.

We learn this early in our Python education: If a function returns multiple values and you don’t use one of them, you indicate this by using _. Here, the next_customer() function returns two values and we’re interested in only one of them:

name, _ = next_customer()

This makes the code more understandable, helps the interpreter generate better code, and tells static code checkers to not warn us about an unused variable.

This is simple. We file this knowledge in the “Python” part of our brain.

Another lowly underscore

The underscore has a recommended use in Django internationalization. The documentation advises us about a convention for aliasing _ to a frequently used i18n function.

It addresses some of the other uses for _:

The underscore character (_) is used to represent “the previous result” in Python’s interactive shell and doctest tests. Installing a global _() function causes interference. Explicitly importing ugettext() as _() avoids this problem. [source]

We take comfort that we’re using _ exactly as the Django examples do. Like this one:

from django.utils.translation import ugettext as _
from django.http import HttpResponse
def my_view(request):
 output = _(“Welcome to my site.”)
 return HttpResponse(output)

This is simple. We file this knowledge in the “Django” part of our brain.

We add i18n support to our codebase

One day we started adding internationalization and localization support to our codebase. We did this in stages using multiple pull requests.

All was well for a couple of weeks. But then a code merge caused a nonsensical exception to start happening in unremarkable code.

The code loaded a message string into a variable for later use:

msg = _(“innocuous string that wouldn’t upset anyone”)

It started raising exceptions that a string was not callable. (?) But this line of code hadn’t changed in weeks. (??)

The codebase changes made circa this time weren’t nearby (???) and were very simple. (????) They deleted unused code and fixed some pylint errors. (We’ve recently started using pylint and are working our way to a 10/10 score.)

We scratched our heads for some minutes. And then understood what had happened!

The not so lowly underscore

One of the pylint fixes replaced an unused variable with _. The code had changed from:

bagel, profile = best_match()

into:

bagel, _ = best_match()

This reassigned the value of _. When it was used to translate a string later in the function, the _() call failed.

This is obvious once you think about it:

>>> from django.utils.translation import ugettext as _
>>> _
<function ugettext at 0x1024666e0>
>>> _ = range(3)[0]
>>> _(“translate this string”)
Traceback (most recent call last):
File “<stdin>”, line 1, in <module>
TypeError: ‘int’ object is not callable
>>>

This bug’s funny characteristic is that it can lay hidden in your code for a long time — possibly forever — without causing a problem. If a function uses _ only for unused variables, or only for translation, or doesn’t translate a string after _ is used for a value, it’ll never cause a problem.

You’ll be bitten by this bug only if code assigns to it and then translates a string within the same function.

We weren’t aware of this bug entering our codebase for four reasons:

1. We read meaning into the Django documentation that wasn’t there. It’s correct but we read it carelessly!
2. Our Python brains signed off on the Python use of _
3. Our Django brains signed off on the Django use of _
4. Our Python brains didn’t talk to our Django brains

The solution is to import ugettext (or one of its cousins) simply, and use it explicitly. Or to not assign to _.