Functional Pipe in Go

Practical Function Composition in Go

Published in

The Startup

7 min readDec 27, 2019

This article introduces a practical approach for implementing the function composition in Go. If you are too impatient to read the whole story, you can directly go to the source code.¹

Introduction

One of the most interesting features of functional programming languages is the ease of composing functions. The function composition in mathematical notation is simply an operator dot ‘⋅’ that takes two functions f(x) and g(x) and produces another function h(x) = (f ⋅ g)(x), defined as f applied to the result of applying g to x, or (f ⋅ g)(x) = f(g(x)).² A pipe ‘|’ operator is also a widely accepted notation in programming which works the same but usually in a different direction, i.e. (f ⋅ g)(x) = (g | f)(x).

Function composition is quite popular in functional programming. So that almost all functional languages have defined a specific operator to make composing functions yet easier. For example, Haskell uses dot . notation.³ Here is an example of function composition in Haskell:

inc :: Integer -> Integer
inc x = x + 1sqr :: Integer -> Integer
sqr x = x * xsqrPlusOne :: Integer -> Integer
sqrPlusOne = inc . sqr

And, F# uses a composition >> or pipe|> notation:⁴

let inc x = x + 1
let sqr x = x * xlet result = (sqr >> inc) 8     // result = 65
let result = (8 |> sqr) |> inc  // result = 65, ()'s are optional

The elegance of composition comes from expressing complicated algorithms by combining much simpler formulas. In functional programming, these formulas are functions, and the result of composition is just another—but more complex—function.

Besides the mathematical interpretation, composition and pipe can represent a broader approach. In real-life programming, we typically encounter business problems which consist of several steps that each step depends on the outcome of the previous step. In their nature, composition and pipe are very beneficial and can provide a general solution for this kind of problem.

The goal of this article is to find a way to bring the power of function composition into the Go world using pipes.

Functional Go

Functions are first-class objects in Go that is crucial to apply functional principles. Unfortunately, Go (as of Go 1.x) lacks a notation for function composition or pipe. However, we can always achieve the same goal by applying functions consequently:

func sqrPlusOne(x int) int {
  sqr := func(x int) int { return x * x }
  inc := func(x int) int { return x + 1 }  return inc(sqr(x))
}

But, the problem arises with Go functions returning multiple values, or commonly, errors as their last return value. In such cases, we need to handle errors explicitly for each function:

func sqrPlusOne(x int) (int, error) {
  sqr := func(x int) (int, error) {
    if x < 0 {
       return 0, errors.New("x should not be negative")
    }
    return x * x, nil
  }
  inc := func(x int) int { return x + 1 }  y, err := sqr(x)
  if err != nil {
    return 0, err
  }  return inc(y), nil
}

This is somewhat straightforward for two functions. But it can quickly become a burden when the number of functions increases.

Solution

Here, we need to find a solution both to compose functions easily and to handle errors transparently. Since the number of function arguments and return values can vary, one can only think of one answer: reflection⁵.

Go’s reflection (via reflect package) is a powerful tool which enables us to investigate functions for their arguments and return values, and to call them dynamically. This is not a tutorial about Go reflection, but here we give you some examples:

sqr := func(x int) (int, error) {
  if x < 0 {
    return 0, errors.New("x should not be negative")
  }
  return x * x
}t := reflect.TypeOf(sqr)    // t.Kind() == reflect.Func
t.NumIn()                   // num inputs:   1
t.NumOut()                  // num outputs:  2
t.In(0)                     // input types:  int
t.Out(0), t.Out(1)          // output types: int, errorv := reflect.ValueOf(sqr)
v.Call(...)                 // dynamic dispatch

As Go does not support overloading and/or defining new infix operators, we have to implement the pipe operator as a function. We start by defining Pipe function and Pipeline type for its result:

type Pipeline func(...interface{}) errorfunc Pipe(fs ...interface{}) Pipeline {
  return func(args ...interface{}) error {
    ...
  }
}

Pipe uses variadic arguments to take zero or more functions as inputs and produces a Pipeline.

A Pipeline instance is just another function that does the actual work. It accepts zero or more inputs and gives an error. The number of its input arguments must match the input arguments of the first function in fs. But its output may or may not match the last function, and that is because Go does not have variadic return values. We call the last function a sink, i.e. its job is to gather the results and clean the pipeline. The sink should not return any values other than an optional error.⁶

With these two definitions, we can rewrite the previous sqrPlusOne example function as:

func sqrPlusOne(x int) (int, error) {
  var result int  err := Pipe(
    func(x int) (int, error) {
      if x < 0 {
        return 0, errors.New("x should not be negative")
      }
      return x * x, nil
    },
    func(x int) int { return x + 1 },
    func(x int) { result = x },  // the sink
  )(x)  // the execution of pipeline  if err != nil {
    return 0, err
  }  return result, nil
}

Here you see how easy it is to compose functions with Pipe. There is no need to handle errors for each function, and the list of composing functions can go on indefinitely.

How can we implement the pipeline? We start by processing the input arguments. Go reflection enables us to call functions dynamically, but for that, we need to pass the input arguments as an slice of reflect.Value values. So first, we need to do the conversion:

var inputs []reflect.Value
for _, arg := range args {
  inputs = append(inputs, reflect.ValueOf(arg))
}

Secondly, we have to solve the nested function calls, f(g(…)). We can always unwind nested functions using a for-loop as following pseudo-code:

// equivalent to f_n(...f_1(x))
for f in fs { x = f(x) }

The same implementation using reflection would look like as:

for _, f := range fs {
  inputs := reflect.ValueOf(f).Call(inputs)
}

But we are missing the errors which should be checked after each function call and should not be passed to the next function. So we extend the previous code and write:

var errType = reflect.TypeOf((*error)(nil)).Elem()for _, f := range fs {
  v := reflect.ValueOf(f)
  t := reflect.TypeOf(f)  outputs := v.Call(inputs)  inputs = nil  // clear inputs  for i, output := range outputs {
    if t.Out(i).Implements(errType) {
      if !output.IsNil() {
        return output.Interface().(error)
      }
    } else {
      inputs = append(inputs, output)
    }
  }
}

And it ends what we require to implement the pipe operator. See how we check return values for errors and neglect the nil errors. A complete implementation of this approach can be found here.

Update: Performance Issues

In order to measure the performance of the previous implementation, we can write a simple benchmark test to compare Pipe against direct function call for a small example:

var result int  // to prevent compiler optimizationfunc BenchmarkPipe(b *testing.B) {
  sqr := func(x int) int { return x * x }
  inc := func(x int) int { return x + 1 }  x := rand.Intn(1000)  b.Run("Direct", func(b *testing.B) {
    for n := 0; n < b.N; n++ {
      result = inc(sqr(x))
    }
  })  b.Run("Pipe", func(b *testing.B) {
    pipe := Pipe(sqr, inc, func(x int) { result = x })    for n := 0; n < b.N; n++ {
      pipe(x)
    }
  })
}

The benchmark on my machine⁷ shows that: while the direct function call results 4–5 ns/op, Pipe runs in ~1000 ns/op which is 200–250× worst. This makes it impractical for repetitive small tasks, as it has a relatively big overhead because of the reflection.

Conclusion

This article presents a rather simple method to implement a pipe operator in Go. It works, but it also eliminates type-safety. It casts all arguments into interface{} before the execution of the pipeline, and it uses reflection to apply functions. So, there is no way to check functions and their arguments at compile-time.

Although it is practical, but in my opinion, it is not a Go-ish way to write applications. It would be nicer if Go has a built-in pipe operator and do the static type-check at compile-time.

Pipe implements a dead-simple synchronized pipeline. It is specially useful for medium-sized tasks. If someone needs a concurrent pipeline, Go already provides a better approach using channels.

¹ The source code is hosted on Github and published under MIT licence.
² See Wikipedia page for the definition of function composition.
³ See Haskell wiki for function composition.
⁴ See MSDN Blog by Chris Smith for function composition in F#.
⁵ See The Laws of Reflection blog by Rob Pike.
⁶ If it does, its return values (except error) will be ignored.
⁷ MacBook Pro, Intel Core i5, 8 GB of RAM