Anonymous Functions and Reflection in Go

I was recently browsing Hacker News and came across an article that caught my eye: Lambdas and Functions in Python. The article — I’d recommend reading it for yourself — details how to use Python lambda functions and shows an example of how they might be used to achieve clean, DRY code.

Reading the article, the portion of my brain that gets turned on by slick design patterns nearly had a nerdgasm. Simultaneously, however, the portion of my brain that’s been trained to hate dynamic languages said, “Eww”. A brief aside to offer some context for my feelings of revulsion towards dynamic languages (feel free to skip ahead if you need no convincing):

I used to be a huge fan of dynamic languages (I still am for certain tasks and use them almost every day). Python was my language of choice throughout college; I used it for scientific computing and to quickly put together small, proof-of-concept projects (my personal website used to use Flask). That all changed when I started work in the real world (Qadium) and contributed to my first large Python projects. These projects contained systems responsible for collecting, processing, and enhancing various well-defined data types. The original choice of Python was made for two reasons: 1) It’s what the early employees were comfortable with and 2) it was fast to develop in. By the time I started, we had recently launched our first B2B product offering and some of the early Python developers had moved on. The code left behind had several issues: 1) it was hard to read 2) it was hard to debug and 3) it was near impossible to change/refactor without breaking something. Additionally, the code had very few tests. These issues were the cost of rapidly prototyping the systems that demonstrated the value of our first product. The problems mentioned above were severe enough that the majority of developer time was spent troubleshooting issues and precious little time was spent developing new features or modifying the system to accommodate our ever increasing desire and need to collect and process more data.

To combat these issues, several other engineers and I began on efforts to slowly re-architect and re-write the systems in statically-typed languages (the overall experience was at times similar to what I imagine it would be like driving a car that’s on fire while simultaneously building another car). For the processing systems, we chose to use Java and, for the data collections systems, we choose to use Go. Two years later, I can honestly say that the use of a static language, like Go (which manages to retain a lot of the dynamic feel of something like Python), has helped to resolve many of the issues we were experiencing.

Now I’m sure a good number of readers will grumble something to the effect of, “Dynamic languages like Python are fine…you just need to organize and test them properly.” I won’t strongly contend with the veracity of this statement, but I will say that static languages helped us and are a better fit for the problems our systems are solving. And after supporting and repairing the dumpster fire that our Python-based production systems at times resembled, I can say I won’t be opting to use dynamic languages for large projects any time soon.

So to go back to my original intent in writing this article, I saw a cool design pattern and I wanted to see if I could easily replicate it in Go. If you haven’t read the aforementioned article, I’ve duplicated the problem it solved with lambdas/anonymous functions below:

Let’s pretend that one of your customers asked you to create a program to simulate a “Reverse Polish Notation calculator” that they will install on all their employees’ computers. You accept this work and get the specs for the program:

The program should be able to do all the basic operations (divide, sum, substract, multiply), to find the square root of a number and to square a number. Obviously, you should be able to clear all the stack of the calculator or just to drop the last inserted number.

If you’re not familiar with Reverse Polish notation (RPN), check out wikipedia or the original article.

To start, the author of the previous article offered a functional but highly redundant piece of code. Here it is, ported to Go:

Note: Go doesn’t have a built in stack, so I’ve created my own:

(Aside from, panicking when we Pop when it’s empty and not being thread safe in any way, it gets the job done.)

The solution above is functional, but duplicates a lot of code — specifically, the code to get the arguments/operands being supplied to the operations.

The Python-lambda article makes incremental improvements on this solution, moving the operation functions as lambdas into a dictionary so they can be looked up by name, looking up the number of operands for an operation at runtime, and using common code to supply the operands to the operation. The final python solution is as follows:

The solution adds a bit of complexity over the original solution, but now adding a new operation is as easy as adding a single line! My first thought when seeing this was: How would I do it in Go?

I was aware that Go had Function Literals, and it was a simple matter, as in the Python solution, to construct a map from operation name to operation function. That can be done as follows:

Note: In Go, in order to keep all of our anonymous functions in the same map, we need to use the empty interface, interface{}, type. Every type in Go implements the empty interface (it’s an interface with no functions; all types have at least 0 functions). Under the hood, Go represents an interface as two pointers: one pointing to the value and the other pointing to the type.

One way to determine the underlying type is to switch on.(type). For instance:

This would output the following (excuse the poor grammar):

SQRT takes one operands
AC takes zero operands
+ takes two operands
/ takes two operands
^2 takes one operands
- takes two operands
* takes two operands
C takes zero operands

This motivates a clear way to go about determining the number of operands an operation takes and how to go about replicating the Python solution. But can we do better? Could we make the logic for extracting and supplying arguments to an operation more general? Can we look up the number of operands a function takes and invoke it without using if or switch statements? It turns out that by using reflection through Go’s reflect package we can.

A brief overview of reflection in Go:

In Go, generally if you need a variable, type, or function you define it and use it. However, if you find yourself in a situation where you need to use information only available at runtime or you are designing a system that consumes multiple different types (e.g. the operation functions — they accept different numbers of arguments and are therefore different types), then you may need to use reflection. Reflection gives you the ability to examine, create, and modify different types at runtime. For a more complete overview of reflection in Go and some basics on how to use the reflect package, see The Laws of Reflection blog post.

The following shows an alternative means by which to lookup the number of operands our anonymous functions take using the reflect package:

Similar to switching on .(type), this would output:

^2 has 1 operands
SQRT has 1 operands
AC has 0 operands
* has 2 operands
/ has 2 operands
C has 0 operands
+ has 2 operands
- has 2 operands

And now I didn’t have to hard code numbers based on my knowledge of the function type signature!

Note: The call toNumIn will panic if the value’s Kind (a Kind is not to be confused with a type) is not a Func, so take care when using it as the panic will only occur at runtime.

Inspecting the Go reflect package, we see that it is actually possible to invoke a value where the Kind is Func by passing a slice of Value objects to the Call method. For example, we can do the following:

This, as expected, outputs:

The result is  5

Cool!

We can now write our final solution:

Here we determine the number of operands (line 64), pull operands off the stack (lines 69–72), and invoke the desired operation without using separate functions to handle the different numbers of operands (line 74). And, as with the Python-based solution, adding a new function is as simple as adding a single map entry with an anonymous function to the catalog (line 30).

To summarize, we’ve seen how to use anonymous functions and reflection in Go to duplicate an interesting design pattern. I would caution against the overuse of reflection. Reflection adds complexity and I’ve often seen it used as a work around for bad designs. Additionally, it can move compile-time errors into runtime errors and it can add a decent amount of slow down (coming soon: benchmarks between the two solutions!)—inspecting the source it looks like reflect.Value.Call performs a good amount of preparation work and allocates a new slice for its []reflect.Value result parameters on each call. That said, if performance isn’t an issue — it probably wasn’t a concern in Python land ;), testing exists, and we’re optimizing for keeping the code short and making it easy to add new operations, then reflection could be the way to go.