Something every great engineer should know

Posted on December 20, 2016 by Iain McDonald

Skyscanner’s core values are helpful in encouraging the right choices to maintain the positive culture we’ve built up over the years. One such value is Master, Teach, Learn. It encourages us to become subject matter experts, seek opportunities to learn new things, and ensure we pass on what we’ve learned to colleagues. An opportunity recently presented itself to share what I knew about declarative programming.

Joel Spolsky

Apologies for the potato like quality, but this is a video still from the fireside chat Joel Spolsky (right) had with our CTO, Bryan Dove (left), in our Edinburgh office earlier this summer. We were excited to have one of the earliest engineering bloggers, and someone responsible for Stack Overflow and Trello, share his experience with us in person.

Near the end of the chat, Bryan asked a great question and Joel’s answer surprised me.

Bryan: Are there things that you believe every great engineer should know, that they commonly don’t?

Joel: The idea of a function as a first class object. That was something I was surprised at how few programmers knew.

Joel goes on to describe how this observation was the basis for a blog post back in 2006 called Can Your Programming Language Do This? This surprised me not because I thought declarative programming was a common practice — in all the interviews where I’ve asked a question requiring iteration over a collection, I’ve yet to see a single answer that doesn’t use a for loop. I was surprised because I’d written a blog post along similar lines the previous year.

Skyscanner University

To help us with the Master, Teach, Learn core value, we have an internal training platform called Skyscanner University. This allows us to attend training courses from external presenters, but crucially, it also allows anyone — any employee — to create their own course and teach to whomever wants to attend. In my time here I’ve learned about Docker containers, coaching, leadership… even home brewing — thanks Raymond 😉

I decided to find out by asking in the #engineering channel in Slack, if anyone would be interested in learning more about declarative programming. I was happy to find a dozen or more interested engineers, so I created a short course (with apologies to Mike, one of our resident functional gurus).

Higher Order Functions

Ganglia showing all nodes in the EMR cluster with CPU at 100%

Sharing understanding of higher order functions first occurred when I was a Data Scientist creating Elastic MapReduce clusters in AWS to run large Spark jobs to transform and aggregate loosely structured data — a key constituent of Machine Learning projects. I wouldn’t have known where to begin if it hadn’t been for Jon Skeet and his reimplementation of LINQ, which I worked through several years earlier.

This is one of the key takeaways about declarative programming: I learned to program in a declarative manner by using LINQ in C#, an object oriented language. Most modern, popular languages have a mechanism for first class functions i.e. the ability to store a function reference in a variable as you would an integer or a string. JavaScript needs no special syntax; Python has lambda expressions; C# has Action and Func classes; and, since version 8, even Java has support.

Enabling parallelisation with the Spark framework is just one advantage of knowing how to code in a declarative style. Declarative code is an evaluation of expressions, whereas imperative code is a series of statements that mutate state. The reduction of dependence on state creates code that’s easier to test in an automated manner because less mocking is required; code that’s more stable to ongoing change because it has less dependence on state side effects; code with greater separation of concerns which aids readability and conceptual understanding.

> const square = x => x * x;
undefined
> square(4);
16

But why is the ability to store a function in a variable so valuable? First class functions enable the use of higher order functions i.e. functions that take functions as parameters, either stored in an intermediate variable or written as an anonymous function, and these function parameters are called during the processing of the higher order function.

Thankfully @steveluscher came up with a simpler pictoral representation of what higher order functions, such as map, filter and reduce, are actually doing. Which kicked off a lot of discussion.

Knowledge & Understanding versus Problem Solving

It’s not enough to know what a higher order function is, or even what a few specific ones do. The skill in declarative programming comes from practicing solving problems. Mastering declarative programming takes both knowledge and understanding and problem solving. The slide deck I presented in the Skyscanner University course was only half the story. The fun came from solving a small problem using only higher order functions in a stateless manner.

I share the problem here with an example imperative implementation, in Python, JavaScript and C#. Can you solve this in a declarative style using map, filter, reduce and groupBy? You’ll have to do a bit more web research than those attending the course, but I’ll leave you with one piece of advice: if the first word of the function is return, you’re going along the right lines.

Try to resist if you haven’t solved it yourself, but the solutions can be found here. Best of luck! And if you haven’t already, why not Learn You A Haskell?

Declarative Programming Repo Problem — can you solve it?

About the author

My name is Iain McDonald, an Engineering Manager based in Edinburgh. I started programming by stumbling into BASIC on the Atari 800XL, and have been hooked ever since. I share random code and blog posts.

Learn with us

Take a look at our current job roles available across our 10 global offices.

We’re hiring!