High-Order and Partially Applied Functions in Python

A bit of functional programming for a Pythonista

--

Disclosure: As one who has written a fair amount of Scala, my brain got somehow trained to look for certain functional programming patterns for problems I face. Hence, I assume there are alternatives to what I’m about to suggest. Feel free to comment and enrich other readers' knowledge. :)

High-order functions (HOFs) take functions as parameters and/or return a function.

Partially applied functions are functions where a subset of the original function parameters is fixed, returning the original function with fewer arguments to input.

Python treats functions as first-class citizens, allowing us to make great use of these two functional programming patterns — even together.

Let’s dive in.

High-order functions in Python

You probably already know two built-in Python HOFs.

1. Map

`def add_ten(number):    return number + 10numbers = [1, 2, 3, 4, 5]added_ten = map(add_ten, numbers)assert list(added_ten) == [11, 12, 13, 14, 15]`

`map` is an HOF — it takes any unary function and applies it to every element of an iterable.

2. Filter

`def only_foo(s):    return s == "foo"my_strings = ["foo", "bar", "baz", "foo"]filtered = filter(only_foo, my_strings)assert list(filtered) == ["foo", "foo"]`

`filter` is also a HOF — it takes a predicate function and applies it to filter-in elements of an iterable.

So how do you build one yourself?

3. Building a HOF

Let’s exemplify if by implementing streamlined versions of `map` and `filter`:

`def map(func: callable, iterable: Iterable):    return [func(x) for x in iterable]def filter(func: callable, iterable: Iterable):    return [x for x in iterable if func(x)]`

I must note here that, although `map` and `filter` are a good way to exemplify HOFs, Pythonistas usually don’t use them. Often we see way more list comprehensions out there. That’s because it’s more pythonic (and some might argue they are probably faster due to internal Python optimizations).

So, let’s take a more real-life example:

`def read_parse_and_write_gzip(file_path: Path,                              output_dir: Path,                              line_parser: callable):    with open(file_path, "rt") as in_f:        with gzip.open(output_dir / (file_path.name + "_parsed.gz"), "wt") as out_f:            for line in in_f:                # line_parser can be any function that maps text lines                modified_line = line_parser(line.removesuffix("\n")) + "\n"                out_f.write(modified_line)`

Our `read_parse_and_write_gzip` method allows us to have a bunch of logic packed in a reusable way, while still permitting the flexibility to reuse it with different parsing logics.

For example, we can use it to add a `"foo"` to the end of each line while compressing it to disk.

`folder = Path("/tmp")file = folder / "foo.txt"with open(file, "wt") as f:    f.write("abc\n123\n")# appeding "foo" to each lineread_parse_and_write_gzip(file, folder, line_parser=lambda x: x + "foo")with gzip.open(folder / (file.name + "_parsed.gz"), "rt") as result:    for line in result:        assert line.endswith("foo\n")`

That’s why it’s called a high-order function — because it’s a function of functions.

In our example, we can now reuse the logic for different input files, if we wish to: the file reading, output file name creation based on a certain convention, compression, and writing in a streaming fashion — all this will be reused functionally.

That’s fascinating and opens up several use cases with extreme flexibility. You probably already know about Python decorators. They take a function and also return one. They are superstars HOFs! 😎

Partially applied functions in Python

Suppose you have a function with `N` parameters. In a specific situation, `M `parameters won’t change (where `M <= N`). For example, we want to have `a, b, c, d` fixed and reuse them for several applications, but change `e`:

`def my_func(a, b, c, d, e):  # do and return somethingres1 = my_func("a", "b", "c", "d", "foo")res2 = my_func("a", "b", "c", "d", "bar")res3 = my_func("a", "b", "c", "d", "baz")`

Seems rather error-prone (for instance, if we need to change `"a"` to `"z"`). Can we do better? Probably:

`def my_func(a, b, c, d, e):  # do and return somethinga, b, c, d = "a", "b", "c", "d"res1 = my_func(a, b, c, d, "foo")res2 = my_func(a, b, c, d, "bar")res3 = my_func(a, b, c, d, "baz")`

Still, I’m not satisfied. 😄

Let’s do it more elegantly anyway.

1. Building a partially applied function

`def my_func(a, b, c, d, e):  # do and return somethingfrom functools import partialmy_pa_func = partial(my_func, "a", "b", "c", "d")res1 = my_pa_func("foo")res2 = my_pa_func("bar")res3 = my_pa_func("baz")`

As we can see, the main property of partially applied functions is that when created, they reduce the function arity. In other words, makes it a smaller degree function by reducing the number of arguments.

If programmers say words like “that’s a unary function”, you know now where they came from. An unary function takes one argument, a nullary takes none, and a binary takes two.

And, arity can be reduced.

2. Creating new semantics

The cool thing about partially applied functions is that they also improve semantics.

We might want to take a generic method and make it semantically applied to a specific use. For example, the `head` and the `tail` commands are applications of `slice`.

Consider the following simplified implementation of slice:

`def slice(l: list, start: int, end: int) -> list:  return l[start:end]`

It’s easily observed that `head` and `tail` are partially applied functions of `slice`:

`head = partial(slice, start=0)tail = partial(slice, end=None)s = "abc"assert head(s, end=2) == "ab"assert tail(s, start=1) == "bc"`

Hence, deriving them from `slice` allows us to narrow down semantics.

Combining HOFs and partially applied functions

Well, if we can use them separately, we can surely combine them. Let’s do that by taking our `read_parse_and_write_gzip` HOF from above and applying different parsing logic to the same input file. We’ll do it passing around our `head` and `tail`:

`# input file content = "abc\n123\n"read_parse_and_write_gzip(file, folder, partial(head, end=2))# written gz file content = "ab\n12\n"read_parse_and_write_gzip(file, folder, partial(tail, start=1))# written gz file content = "bc\n23\n"`

Want to go further?

`# input file content = "abc\n123\n"apply_to_my_file = partial(read_parse_and_write_gzip, file_path=file)apply_to_my_file(output_dir=folder, line_parser=partial(head, end=2))# written gz file content = "ab\n12\n"apply_to_my_file(output_dir=folder, line_parser=partial(tail, start=2))# written gz file content = "bc\n23\n"`

Ok, maybe that was too much… 😆 But you get the idea.

Using function programming patterns as high-order and partially applied functions allows for great flexibility and composability. Thankfully, Python supports those two very well.

Happy Python coding! 🚀

I’m a senior Big Data Engineer working at Tikal — a company composed of tech experts, where we provide hands-on consultancy, scaling R&D teams with cutting-edge technologies. Check Tikal’s Tech Radar to learn what technologies we suggest to try, start, keep, and stop.

--

--

Senior Big Data Engineer @ Tikal - Home of Tech Experts