ADVANCED PYTHON PROGRAMMING

A Value by Any Other Name

In this article, we cover assignment, name resolution and scopes. Once and for all—how does it work in Python?

10 min readApr 3, 2020

Python scopes are actually very simple — but for some reason, they were never explained to me in simple terms. It was always about “closures”, “bound variables” and “free variables”, “passing by value” and “passing by reference”, comparisons to Java and C… and for what? The simple truth is that Python scopes are a hierarchy of dictionaries, and if you think of them as such, everything suddenly makes sense.

Objects, Floating in Space

The first thing to understand is that all the values in Pythons are objects, floating in space: not just class instances, but also native types such as numbers and strings, functions, and even classes. These objects have attributes and methods as you’d expect, but generally—they’re floating in space. Python allocates them for you, manages their internal representation, and reclaims their memory when you’re done; for you, they’re just there.

But of course, that’s not enough—you need a way to refer to those values, which is what names are for. The name itself is just a pointer to the object, which exists in and of itself, as we’ve established. This is the simplest way to explain why Python always passes everything by reference: it’s not copying anything, because in Python, assignment doesn’t deal with values—it only deals with names, and simply binds a new name to the same object.

>>> x = [1, 2, 3]>>> y = x
>>> y.append(4)
>>> x
[1, 2, 3, 4]>>> def append_5(items):
...     items.append(5)
>>> append_5(x)
>>> x
[1, 2, 3, 4, 5]

In this example, the value [1, 2, 3] is created and bound to the name x. It’s then also bound to the name y—and instead of thinking about it in terms of “assignment operators” or “copy constructors”, suffice to say we just establish a new way to refer to the same object.

That’s why, when we append 4 to y, x becomes [1, 2, 3, 4]. A more accurate way to phrase it would be that the object becomes [1, 2, 3, 4], whether we call it x or y.

Similarly, when we pass x to the append_5 function, we’re not “passing it by value” or “passing it by reference”—we’re simply invoking a function, which is a unit of parametrized code, and binding our object to its items parameter; the function then appends 5 to it, whatever its name.

Shakespeare wrote that “A rose by any other name would smell as sweet”—similarly, a value by any other name would be the same object, just floating in space.

Namespaces are One Honking Great Idea

The second thing to understand is that all names in Python are managed in dictionaries—or, more generally, namespaces—which are associated with units of code and called “scopes”. Every time you assign a value to a name, Python stores this information in a scope, almost as if it were adding a key to a dictionary: scope[name] = value; and every time you access a value through a name, Python tries to resolve it using the available scopes, like in scope[name]. These namespaces are literally available as dictionaries using the locals() and globals() builtin functions:

>>> x = 1
>>> def f():
...     y = 2
...     def g():
...         z = 3
...         print(locals())
...         print(globals())
...     g()
>>> f()
{'z': 3}
{..., 'x': 1}

When you just start out, whether you’re writing a module or using Python’s interactive interpreter, you start at the global scope; and every time you “indent into” a function or a class, a new scope is created and associated with that unit of code, and units of code nested within it.

So, in our outermost scope, we’ve had the name x bound to the value 1; then, inside f, we had y bound to 2; and finally, inside g, we had z bound to 3. That’s why, when we printed g's local scope, we got a dictionary with only z bound to 3; and when we printed g's global scope, it went all the way up to a dictionary with x bound to 1, and a bunch of other builtins Python puts there by default (which is why utilities such as the len() function or the ValueError class are always available to you—they’re not keyword, but predefined global variables [that you can {but shouldn’t} override]). And if the idea still hasn’t sunk in, this should drive the point home:

>>> x = 1
>>> namespace = locals()
>>> namespace['x']
1
>>> namespace['x'] = 2
>>> x
2
>>> del namespace['x']
>>> x
Traceback (most recent call last):
  ...
NameError: name 'x' is not defined

Once we get our hands on the local scope in its dictionary form, we can manipulate it as such—which immediately reflects on how basic name resolution is done. That’s why it’s a bad idea to compare Python names to C variables, or Python scopes to the program’s stack; it’s an entirely different beast, resolved dynamically, (almost) like a dictionary. It does have several nuances, so let’s take a closer look at name resolution.

Gettin’ It

Consider the following code:

x = 1
def f():
    y = 2
    def g():
        z = 3
        print(z)
        print(y)
        print(x)
        print(w)
    g()

What happens when we call f, which calls g? First, z is printed—but before we can print it, we have to pluck the object out of that space it’s floating in. Luckily, we needn’t go far: z was bound to 3 in the local scope, which is, not surprisingly, the first namespace that Python checks.

Then, y is printed. This is trickier: the first lookup, in the local scope, comes up empty. But since g is nested inside f, whose local scope has y bound to 2, we get to that namespace soon enough. The same goes for x: it’s missing from both g and f, but is found in the local scope of their module—more commonly referred to as the global scope. Finally, w is missing from all three scopes, and since there’s nothing “above” the global scope, Python regrettably raises a NameError. Pretty simple and intuitive, right?

Python is actually smarter than that, and has some optimizations in place, which let it know each name’s scope in advance, and “jump” straight to the right namespace to resolve it. This has to do with closures, bound variables and free variables—but it doesn’t really help with understanding scopes, so we’ll cover it later.

Set and Delete

We talked about name resolution; but what about assignment and deletion? This shouldn’t surprise you:

>>> x = 1
>>> def f():
...     x = 2
>>> f()
>>> x
1

What happens is that x is bound to 1 in the global scope; but inside f, the same name is bound to a different value, 2. Assignment does not perform the lookup that resolution does—otherwise, it’d find that name in the global scope, and change it there. Assignment operates on the local scope right away, even if it means “shadowing” names from enclosing scopes. Similarly:

>>> x = 1
>>> def f():
...     del x
>>> f()
Traceback (most recent call last):
  ...
UnboundLocalError: local variable 'x' referenced before assignment

That’s a weird error message, but it makes sense if you think about assignment and deletion as inherently local: if they only ever operate on the local scope, del x only ever looks in f's namespace, and gets annoyed that the local variable x was referenced (specifically, deleted) before it was properly assigned.

Transcending Scopes

Fear not: there are ways to set and delete names in scopes other than the local one—and while generally frowned upon, it is occasionally useful, and fits well in our model of hierarchical dictionaries.

>>> x = 1
>>> def f():
...     global x
...     x = 2
>>> f()
>>> x
2

The global keyword is actually much simpler than it looks: all it does is instruct Python that in the following unit of code (that’s why it has to be at the beginning of the function), whenever the name x is used, it should go straight to the global scope—for assignment and deletion as well as for resolution. So, setting and deleting values still only ever operates on a single scope; we just “rewire” it, so that specifically for x, this scope is actually the global one.

>>> x = 1
>>> def f():
...     global x
...     del x
>>> f()
>>> x
Traceback (most recent call last):
  ...
NameError: name 'x' is not defined

If we think about it in terms of a hierarchy of dictionaries, Python simply adds a mental note: for this key, it’ll work with that namespace; for others, it’ll work as usual.

Names in Limbo

But what if a name is stuck in limbo, like here:

>>> def f():
...     y = 2
...     def g():
...         # Can we set or delete y?
...     g()
... f()

If we do y = 3, it will bind y to 3 in the local scope; and if we try to delete it, we’ll get an UnboundLocalError like before. If we use the global keyword to mark y as a special case, it’ll jump straight to the global scope, skipping that middle scope we’re interested in.

In Python 2, it was much trickier to get there—but Python 3 introduced the nonlocal keyword, which figures out the closest enclosing scope that has this name (local scope notwithstanding), and proceeds to get, set and delete it from there. Not terribly useful, but yet another piece of the puzzle:

>>> def f():
...     y = 2
...     def g():
...         nonlocal y
...         y = 3
...     g()
...     print(y)
>>> f()
3

Late Binding

Finally, I’d like to tell you about an all-too-common bug that stems from this fundamental misunderstanding of scopes. In it, we create 10 lambda functions in a loop, each returning a different number:

>>> fs = []
>>> for i in range(10):
...     f = lambda: i
...     fs.append(f)

What do you get when you call f(0)? Surprisingly—9. And what do you get when you call f(5)? Also 9. In fact…

>>> i = 1000
>>> fs[0]()
1000
>>> fs[5]()
1000

The truth is, we haven’t created 10 lambda functions that return different numbers; we’ve created 10 identical lambda functions that return i. And since this name is not bound to anything in their local scopes, each and every one of them does the same: keeps looking for it in higher scopes. After the loop ends, i stays bound to the value of the last iteration, 9; so that’s what all the functions find; and when I manually change it to 1000, the functions seem to change to reflect this change—when in fact, they’re doing exactly the same as before: return globals()[i]. An easy way to solve it would be to introduce an intermediary scope:

>>> def create_function(i):
...     return lambda: i>>> fs = []
>>> for i in range(10):
...     f = create_function(i)
...     fs.append(f)>>> fs[0]()
0
>>> fs[5]()
5

In this case, every iteration invokes create_function, which creates a different scope. In the first iteration, this scope’s i is bound to 0; in the sixth iteration, to 5. The functions’ code is still essentially the same: “return the non-local name i”; the difference is that each function is enclosed in a different scope, often called a “closure”, so the hierarchy of dictionaries it traverses in an attempt to resolve a name is different. In the first iteration, its enclosing scope—that is, create_function's local scope—is {'i': 0}, so it returns 0; in the sixth iteration, it’s {'i': 5}, so it returns 5.

As an aside, there’s another, dirtier way to solve this “late binding” issue—introduce i to the functions’ local scopes. At first, this sounds impossible; even if we swap them for regular functions that support assignment statements, we’d end up with this:

>>> fs = []
>>> for i in range(10):
...     def f():
...         i = i # ???
...         return i
...     fs.append(f)

However, the local scope of a function includes not only the names it defines in its code, but also its parameters—and we can use that to inject a value into it via a default argument.

>>> fs = []
>>> for i in range(10):
...     f = lambda x=i: x
...     fs.append(f)>>> fs[0]()
0
>>> fs[5]()
5

In this case, the functions’ code is still the same: return locals()['x']. But in the first iteration, when the function is defined and the default argument for x is evaluated, it’s pointing to the value 0—so that’s the value that gets associated with the parameter x. The sixth time around, however, it’s pointing to 5, and that’s what’s retained.

Conclusion

That was… a lot of words, to describe something I claimed to be “actually very simple”—but most of it was establishing some context, and giving a lot of examples. If I had to sum it all in one sentence, it’d be: Python values are objects, floating in space, and referred to by names, which are stored in a hierarchy of dictionaries; name resolution traverses this hierarchy, from the local scope to the global scope, while assignment and deletion work on a single scope—usually the local one, but this can be changed with the global and nonlocal keywords. Understanding this will make your life as a Python developer much simpler—and come in handy in the future, when we talk about attribute and method resolution.