“You don’t need a macro”

Except when you do

It’s a thing you commonly hear in programming communities dedicated to languages that have macros. “Macros are the tool of last resort”, they say. “You never need a macro”.

I think that macros are misunderstood as a result of having been misused by misguided souls who used macros to solve problems that could have been solved by normal language constructs. Programmer communities are surprisingly susceptible to the proliferation of memes, and so the anti-macro sentiment became a meme as a result of a few bad experiences at some point in the past.

My friend and former colleague Brendan McAdams inspired me recently to learn Scala macros. As is common, Brendan gave a thought-provoking talk (PDF) that, well, provoked some thought. You see, previously I was one of those people who unthinkingly propagated the anti-macro meme, despite having had zero personal experience with macros — in Scala or anywhere. So, as Brendan’s slides rolled by on my screen, I decided to set out on a macro adventure of my own, in search of truth regarding macros in Scala.

After a couple of weeks spent writing, debugging and reading macro code, I’m coming to the conclusion that there are only two categories of problems that can benefit from macros:

  • Class A: Grievous holes in the Scala language itself, or mere omissions in the Scala standard library.
  • Class B: New language constructs that do not and possibly can not exist in vanilla Scala.

This conclusion crystallized in my mind while I wrote two macro-based projects. One — a better enumeration library for Scala called “numerato” — falls mostly in the Class A space. The other — a event sourcing/CQRS library that I haven’t formally published yet — is squarely within Class B.

In this post we’ll take a look at numerato in an attempt to understand the problem it tries to solve, as well as the role of Scala macros in implementing the solution.

A grievous hole

Anyone coming to Scala from Java will be unpleasantly surprised by the absence of a good enumeration facility in Scala. Java has one, yet Scala offers “only” ADTs, which are powerful enough, but are a poor tool to use in emulating a proper enumeration. ADTs are great for creating hierarchies of values that can be sealed against intrusion from outside (i.e., a new value cannot be introduced post-definition), and provide a small degree of safety in verifying exhaustiveness of matches. Scala’s compiler will helpfully emit warnings when a match involving a sealed type hierarchy omits some of the possible values.

Java enumerations, however, go a bit further than this. A Java enum provides facilities for enumerating (no pun intended) the enum’s values at runtime by calling the static value() method. Another useful feature is being able to convert an enum value to and from its integer ordinal index. Java enums even edge into ADT-like territory by allowing enum values to carry custom fields and methods. I think we can all agree that — used properly — a Java enum can be quite powerful.

Scala offers a much more limited enumeration as part of its standard library. The astute reader will already notice a crucial aspect of Scala enumerations, one that makes them a lot less useful than Java’s: in Java, enumerations are part of the language, whereas in Scala, a mere stdlib feature implements an enumeration-like half-solution that is awkward to use and is not very useful to begin with.

Critically speaking, my main beef with both Scala ADTs and enumerations is that there is no way to guarantee — and I use this word in the strongest possible sense — at compile time that a match involving an enumeration or ADT will absolutely, positively be exhaustive. I say guarantee because I have found through painful experience that a mere warning is often woefully insufficient. It is my firm belief that discipline is great, but enforcement of discipline that itself relies on more discipline is doomed to fail.

Filling the hole

So, I set out to dream up a better Scala enumeration facility. I started with the syntax:

Here we’ve defined an enumeration named “Status”, with two possible values: “Enabled” and “Disabled”. To ease transition from traditional Scala enumerations, I’ve decided to keep the declaration syntax.

In case you’re wondering, a Java equivalent of the above idealized enum would look something like this:

Considering these two code snippets side by side, I think it’s pretty obvious that we’re attempting to create a missing language feature, making our quest fall squarely in Class A territory.

Our new enum implementation must meet the following goals:

  • Always warn about non-exhaustive matches in the same way Scala’s regular pattern matching does.
  • Provide an optional stricter matching facility that will emit a compiler error when it encounters an incomplete match.
  • Allow run-time iteration over enumeration values.
  • Provide lookup by ordinal index as well as enumeration name, which we can simply infer for the time being.
  • Allow enumeration types to carry fields in addition to the automatically provided “index” and “name”. These fields must (obviously!) be immutable & publicly addressable via every enum instance.

With this in mind we can start laying down the foundation.

Filler material

What would it take to hit the proposed goals? Let’s start with the first: we already know that Scala’s own exhaustiveness checking relies on a sealed type hierarchy. Armed with this knowledge we can rewrite our idealized enum syntax in plain Scala:

Next, we promised to provide an ordinal index and readable name on each enum value. This is also easy to achieve:

Finally - and before we tackle exhaustive matches - we can add the promised iteration and lookup functionality:

We’ve gradually translated a sort of idealized syntax implementing a missing Scala feature into a naive-but-workable implementation. There’s only one problem with this: we’ve done the work manually, while Java does this automatically on the programmer’s behalf.

Thus our goal becomes to automate the above process of implementing a language feature. This is where macros come in.

Abstract syntax tree hugger

In technical terms, the transformation of code we’ve effected above is just another AST transform: we’ve taken one Scala syntax tree and produced another. This is exactly what macros are for; in fact, macro inputs and outputs are all in the form of Scala ASTs.

Our task is thus to write code which will transform the above “class Status { … }” declaration into some AST representing the final result we’ve produced manually. If this sounds familiar, that’s because it is: we need to write a “function” of sorts, which will take a Scala AST representing “class Status { … }”, and will return another AST representing the transformed result.

Such a function can take the form of a macro annotation. Our idealized syntax thus becomes:

Let’s go ahead and lay down the boilerplate foundation for this “@enum” annotation:

See that “def decl(annottees: Tree*): Tree”? This is our AST transformation function: that’s where we’ll do all of the work.

Let’s think back to our idealized syntax: on some level, we can think of it as just a bunch of text containing pieces we want to extract, then re-arrange and otherwise re-use in making some new piece of text, er, I mean code. It’s almost like turning some text into a template-ized version of itself. This intuition is very helpful in writing Scala macros because of one killer feature: quasiquotes.

Here I encourage you to view (or review) the slides from Brendan’s talk. He gives a good explanation of quasiquotes there, which boils down to “AST templating”, and wouldn’t be as clear coming from me.

So, armed to the teeth with quasiquotes, we can write down a template-like structure that will allow us to extract useful pieces from the input AST, and then construct a new AST containing all the new generated goodness:

Quasiquotes allow us to use the exact same syntax to both pattern-match against an existing AST, and produce a new AST afterwards. The upshot is that we can write macro code that’s easy on the eyes & doesn’t hide any treachery behind a veil of simplicity.

Better still is the fact that whatever AST we produce will be passed through the “typer” phase, which means that mistakes we make in generated code will be caught by the compiler, almost as if the code was written manually to begin with.

Without further ado I present the generated code:

Well, it looks kind of ugly, but I’m glad I don’t have to type this in by hand for EVERY. SINGLE. ENUM.

Fill the hole, seal the leaks

Emboldened by this newfound power of creation^W code generation, we can now correct some problems with our first generated enum. While we had the presence of mind to mark the base enum type “sealed” to prevent rogue derivation from this type in other compilation units (aka files), nothing prevents the intrepid explorers among our users from doing all sorts of silly things, such as:

This will, unfortunately, compile. As you can imagine, “JokeIsOnYou” shouldn’t exist, yet it does, and makes our enum essentially useless, since neither “Status.values()” nor the lookup methods know anything about “JokeIsOnYou”. What can we do?

Since we’re simply writing Scala “with templates”, all we have to do is create a protected abstract type within the companion object we generate for each enum type, then require a value of this protected type to be passed in every time an enum value derives from the base generated enum type. This is best illustrated like so:

Here, “Sealant” is the type we hide within our generated companion. The only possible instance of “Sealant” exists inside the generated companion, which also happens to be the only place that subtypes of our enum - i.e. all of its values - can be instantiated. The result is what we expect; deriving from “Status” outside of the macro-generated companion is no longer possible:

This is kind of fun! What else can we do?

Switching gears

We’ve sealed up our enum as tightly as we could, ensuring that Scala’s own exhaustiveness check will emit a warning in case we forget to account for every enum value in a pattern match. However, as mentioned previously, warnings are way too easy to ignore. Can we create an optional syntax that will emit an error instead of a warning? Let’s shoot for this:

This looks pretty good, and it’s easy to implement with another macro. To start, let’s position the “switch” method to hang off our generated companion:

Here we’re taking a partial function capable of maybe converting “Status” to some “A”, and we turn it into a total “Status => A” function using a macro. This one’s a bit more involved than straight-up quasiquotin’, so feel free to study its implementation on your own time: https://github.com/maxaf/numerato/blob/bb834f35cf2eb53f4b75e375676f64fbe52ea708/src/main/scala/switch.scala

Let’s illustrate the “switch” syntax at work; suppose we had gone ahead and added another value - say, “Unknown” - to our “Status” enum. The compiler now emits an error when it sees that we’ve forgotten to include the newly added enum value in our pattern match:

Yes! It blows up at compile time. Take that, Java!

Parting thoughts

We’ve written a couple of macros that implement a saner, safer, easier to use enumeration for the Scala language. Whether or not you agree that this is a missing language feature is unimportant. The purpose of this exercise was to illustrate a valid use case for macros as a tool for creating new syntactical constructs.

As mentioned previously, material for this post was taken from a library called “numerato”, which makes it easy to take advantage of the “@enum” annotation and “switch” syntax.

Stay frosty & happy quasiquotin’!