Beyond the For-Loop: Higher Order Functions and List Comprehensions in Python

A higher order function is a fuction that operates on other functions, either by taking a function as its argument, or by returning a function. Since functions are first-class objects this is easy to do in Python, and the standard library comes with several higher order functions that make list operations more convenient. Let’s take a look at some of those.

Filter

Let’s say you have a list of integers and you only want the numbers that are even.

We could solve this with a for-loop:

integers = range(0, 10)  
even = [] 
for i in integers:     
    if i % 2 == 0:         
        even.append(i)  
>>> even 
>>> [0, 2, 4, 6, 8]

However, Python has a built in higher order function which can do the same more succinctly — filter(). It takes a function and an iterable as arguments and constructs a new iterable by applying the function to every element in the list.

even = filter(lambda x: x % 2 == 0, integers)

We have reduced the for-loop’s four lines to one. Note that we use a lambda function, an anynomous function, that can be convenient for short ad-hoc functions. We could also have written:

def is_even(x):     
    return x % 2 == 0  
even = filter(is_even, integers)

In addition to higher order functions Python has list comprehensions, so the above could also be written as:

even = [x for x in integers if x % 2 == 0]

What’s going on here? We simply say that we want each integer in integers if it is divisible by 2. It is in essence a flattened, more succint for loop. We don’t have to call any functions.

One problem, three solutions. How do you know which one to use? It comes down to a matter of taste, speed and readability.

The for-loop is the longest, but it’s also easier to read and understand for someone new to Python or higher order functions. For someone who knows Python well the other versions might be both easier to read and write since they are shorter.

The higher order function and the list comprehension are both one-liners. Seasoned Python programmers tend to prefer list comprehensions. They state their intent clearly and can be read from left to right. In this case the list comprehnsion solution is also faster, since it doesn’t make any function calls.

Map

Map applies a function to every element in a list. Unlike filter, it leaves the number of elements in the list unchanged. Let’s say we want to square every element in a list (since map() and filter() return iterators in Python 3, we surround map() with list() to get a readable output):

integers = range(0, 10)
list(map(lambda x: x * x, integers))
>>> [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

And as a list comprehension:

integers = range(0, 10)
[x * x for x in integers]
>>> [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

Again, the list comprehension is faster, and I would argue it is also more convenient and easier to read once the notation is familiar.

Reduce

So far we’ve looked at functions that both consume and return lists. There are also numerous functions that consume a list but return a single value.

Reduce works by applying a function that takes two arguments to the elements in the list, from left to right. Here we use it to get the product of all the integers in the list:

from functools import reduce
integers = range(1, 10)  
reduce(lambda x, y: x * y, integers)  
>>> 362880

What about using a list comprehension here? We can’t. List comprehensions always returns iterables, not single values.

Reduce is very powerful, but can also be hard to comprehend. Notice the import statement—reduce() has been ostracized from Python’s core functions and had to be imported. In fact, Guido van Rossum hates reduce(). In his own words:

“So now reduce(). This is actually the one I’ve always hated most, because, apart from a few examples involving + or *, almost every time I see a reduce() call with a non-trivial function argument, I need to grab pen and paper to diagram what’s actually being fed into that function before I understand what the reduce() is supposed to do. So in my mind, the applicability of reduce() is pretty much limited to associative operators, and in all other cases it’s better to write out the accumulation loop explicitly.”

Luckily Python includes a handful of functions that could all have been implemented by reduce(), but are convenient to have as stand-alone functions. Since they neither consume or return functions, they are not higher order functions.

Sum

Sum simply adds all the elements in a list:

integers = range(1, 10)
sum(integers)  
>>> 45

Any and All

Any and all works just like the names suggest:

any([False, True, False])   
>>> True    
all([False, True, False])   
>>> False

The all() and any() functions are good examples of how functions that take an iterator can be combined with list comprehensions. To check for even numbers in a list:

integers = range(1, 10)
any(x % 2 == 0 for x in integers)   
>>> True
all(x % 2 == 0 for x in integers)   
>>> False

When not overused this combination can result in succinct, readable and Pythonic code.

Are We Functional Now?

Not really. Like we mentioned, Guido van Rossum was never a fan, and Python lacks the enforcement of both immutable data structures and pure (side effect-free) functions you see in more strictly functional languages.

But in line with Python’s pragmatic approach, the languge has assimilated some useful concepts. Personally I think list comprehensions work so well in Python I seldom reach for map() or filter().

Check out Python’s built in functions for other higher order functions. Also, take a look at Python’s itertools library.