High-Order and Partially Applied Functions in Python
A bit of functional programming for a Pythonista
Disclosure: As one who has written a fair amount of Scala, my brain got somehow trained to look for certain functional programming patterns for problems I face. Hence, I assume there are alternatives to what I’m about to suggest. Feel free to comment and enrich other readers' knowledge. :)
High-order functions (HOFs) take functions as parameters and/or return a function.
Partially applied functions are functions where a subset of the original function parameters is fixed, returning the original function with fewer arguments to input.
Python treats functions as first-class citizens, allowing us to make great use of these two functional programming patterns — even together.
Let’s dive in.
High-order functions in Python
You probably already know two built-in Python HOFs.
1. Map
def add_ten(number):
return number + 10
numbers = [1, 2, 3, 4, 5]
added_ten = map(add_ten, numbers)
assert list(added_ten) == [11, 12, 13, 14, 15]map is an HOF — it takes any unary function and applies it to every element of an iterable.
2. Filter
def only_foo(s):
return s == "foo"
my_strings = ["foo", "bar", "baz", "foo"]
filtered = filter(only_foo, my_strings)
assert list(filtered) == ["foo", "foo"]filter is also a HOF — it takes a predicate function and applies it to filter-in elements of an iterable.
So how do you build one yourself?
3. Building a HOF
Let’s exemplify if by implementing streamlined versions of map and filter:
def map(func: callable, iterable: Iterable):
return [func(x) for x in iterable]
def filter(func: callable, iterable: Iterable):
return [x for x in iterable if func(x)]I must note here that, although map and filter are a good way to exemplify HOFs, Pythonistas usually don’t use them. Often we see way more list comprehensions out there. That’s because it’s more pythonic (and some might argue they are probably faster due to internal Python optimizations).
So, let’s take a more real-life example:
def read_parse_and_write_gzip(file_path: Path,
output_dir: Path,
line_parser: callable):
with open(file_path, "rt") as in_f:
with gzip.open(output_dir / (file_path.name + "_parsed.gz"), "wt") as out_f:
for line in in_f:
# line_parser can be any function that maps text lines
modified_line = line_parser(line.removesuffix("\n")) + "\n"
out_f.write(modified_line)Our read_parse_and_write_gzip method allows us to have a bunch of logic packed in a reusable way, while still permitting the flexibility to reuse it with different parsing logics.
For example, we can use it to add a "foo" to the end of each line while compressing it to disk.
folder = Path("/tmp")
file = folder / "foo.txt"
with open(file, "wt") as f:
f.write("abc\n123\n")
# appeding "foo" to each line
read_parse_and_write_gzip(file, folder, line_parser=lambda x: x + "foo")
with gzip.open(folder / (file.name + "_parsed.gz"), "rt") as result:
for line in result:
assert line.endswith("foo\n")That’s why it’s called a high-order function — because it’s a function of functions.
In our example, we can now reuse the logic for different input files, if we wish to: the file reading, output file name creation based on a certain convention, compression, and writing in a streaming fashion — all this will be reused functionally.
That’s fascinating and opens up several use cases with extreme flexibility. You probably already know about Python decorators. They take a function and also return one. They are superstars HOFs! 😎
Partially applied functions in Python
Suppose you have a function with N parameters. In a specific situation, M parameters won’t change (where M <= N). For example, we want to have a, b, c, d fixed and reuse them for several applications, but change e:
def my_func(a, b, c, d, e):
# do and return something
res1 = my_func("a", "b", "c", "d", "foo")
res2 = my_func("a", "b", "c", "d", "bar")
res3 = my_func("a", "b", "c", "d", "baz")Seems rather error-prone (for instance, if we need to change "a" to "z"). Can we do better? Probably:
def my_func(a, b, c, d, e):
# do and return something
a, b, c, d = "a", "b", "c", "d"
res1 = my_func(a, b, c, d, "foo")
res2 = my_func(a, b, c, d, "bar")
res3 = my_func(a, b, c, d, "baz")Still, I’m not satisfied. 😄
Let’s do it more elegantly anyway.
1. Building a partially applied function
def my_func(a, b, c, d, e):
# do and return something
from functools import partial
my_pa_func = partial(my_func, "a", "b", "c", "d")
res1 = my_pa_func("foo")
res2 = my_pa_func("bar")
res3 = my_pa_func("baz")As we can see, the main property of partially applied functions is that when created, they reduce the function arity. In other words, makes it a smaller degree function by reducing the number of arguments.
If programmers say words like “that’s a unary function”, you know now where they came from. An unary function takes one argument, a nullary takes none, and a binary takes two.
And, arity can be reduced.
2. Creating new semantics
The cool thing about partially applied functions is that they also improve semantics.
We might want to take a generic method and make it semantically applied to a specific use. For example, the head and the tail commands are applications of slice.
Consider the following simplified implementation of slice:
def slice(l: list, start: int, end: int) -> list:
return l[start:end]It’s easily observed that head and tail are partially applied functions of slice:
head = partial(slice, start=0)
tail = partial(slice, end=None)
s = "abc"
assert head(s, end=2) == "ab"
assert tail(s, start=1) == "bc"Hence, deriving them from slice allows us to narrow down semantics.
Combining HOFs and partially applied functions
Well, if we can use them separately, we can surely combine them. Let’s do that by taking our read_parse_and_write_gzip HOF from above and applying different parsing logic to the same input file. We’ll do it passing around our head and tail:
# input file content = "abc\n123\n"
read_parse_and_write_gzip(file, folder, partial(head, end=2))
# written gz file content = "ab\n12\n"
read_parse_and_write_gzip(file, folder, partial(tail, start=1))
# written gz file content = "bc\n23\n"Want to go further?
# input file content = "abc\n123\n"
apply_to_my_file = partial(read_parse_and_write_gzip, file_path=file)
apply_to_my_file(output_dir=folder, line_parser=partial(head, end=2))
# written gz file content = "ab\n12\n"
apply_to_my_file(output_dir=folder, line_parser=partial(tail, start=2))
# written gz file content = "bc\n23\n"Ok, maybe that was too much… 😆 But you get the idea.
Using function programming patterns as high-order and partially applied functions allows for great flexibility and composability. Thankfully, Python supports those two very well.
Happy Python coding! 🚀
I’m a senior Big Data Engineer working at Tikal — a company composed of tech experts, where we provide hands-on consultancy, scaling R&D teams with cutting-edge technologies. Check Tikal’s Tech Radar to learn what technologies we suggest to try, start, keep, and stop.
