Testing combinatorial cases with Elixir macros

13 Feb 2017

In this short article we’ll see how to leverage Elixir macros, and a little ExUnit trick, to easily and safely test our application. This method is especially useful when we have a set of cases that can be combined in different orders or groups. What I usually see happening is that the developer will write a couple of tests for a few simple cases and call it a day. Sometimes this is good enough, sometimes it can lead to trouble down the road.

From a true story…

I had been tasked to update our interface to a business-critical third-party service. While under normal circumstances this would mean reading the new docs, checking that the integration tests are still green, having QA test for non-regression, these were not normal circumstances.
All we knew about the new API was that the WSDL of their SOAP service changed, some endpoints instead of being called “abc” were now called “abcv2”. Some redundant parameters were also removed from the requests. The last docs they issued were from 2 years ago and they didn’t have a reliable testing environment. Luckily all the changes were on GET endpoints.

After making the appropriate changes in the code, what surprised me the most was that all the tests were still green. I had to dig deeper. I decided to make a list of all the calls and diff the responses from the new and old endpoints. The new API had an extra field and another one was missing, that was all.

Testing karma

Checking manually was too cumbersome: the order of the fields in the respective responses was different, and the new responses had a new field we couldn’t care less about. Clearly there was an easy way to automate this process and leave a little gift to my colleagues in the form of a test (or 400!).

My idea was to loop through all the calls and parameters, call the old and new API, decode the responses in a %Map and assert their equivalence. Decoding the responses to a Map would solve the issue with the order and the extra field: ExUnit shows you a green\red diff in the terminal. I was OK with a few failing asserts since the new API has a new field and the service I’m calling is far from deterministic.

Property-based testing wasn’t good enough for a few reasons. First of all, I’m not testing for a property of a function, this is an integration test. Secondly, the space of all possible cases is small enough to be tested exhaustively. Lastly, some tests will fail and I’m OK with that, as long as I can see how.

Macros to the rescue

At this point I knew that macros were the tool I needed. I had read a little book on Elixir macros a few months prior, but I was hardly confident with them. After a few hours of tinkering and googling I had an almost working macro. The remaining piece of the puzzle was having ExUnit register an appropriately named test for each function the macro was generating. Luckily I had the chance to use a property-based testing library written by a colleague, so I knew this was possible. More studying, more tinkering, eventually this is what I came up with:

test/support/my_macro.ex

defmodule Service.Support.MyMacro do

  @relative_urls [":param/abc", ":param/def"]
  @params ["one", "two"]

  defmacro non_regression_test do
    for url <- @relative_urls, param <- @params do
      stub = String.replace(url, ":param", param)
      quote bind_quoted: [stub: stub] do
        test_func_name = ExUnit.Case.register_test(__ENV__, :test, stub, [[args: stub]])
        def unquote(test_func_name)(%{args: stub}) do
          assert call_local(stub) == call_remote(stub)
        end
      end
    end
  end
end

test/non_regression_test.exs

defmodule Service.NonRegressionTest do

  use Service.ConnCase
  import Service.Support.MyMacro

  non_regression_test()

  def call_local(call) do
    ...
  end

  def call_remote(call) do
    ...
  end
end

This simple macro will:

• loop through all the combinations of relative urls and params
• ask ExUnit to register a function as a test
• generate such function

Consider @relative_urls to be the list of endpoints under test and @params a list of arguments for said calls. The initial for will loop through all the parameters for each URL. The macro magic starts from quote. For the uninitiated, quote will take some Elixir code and transform it into an AST (Abstract Syntax Tree), which is the underlying structure that the interpreter will eventually evaluate for every Elixir program. bind_quoted will make sure that the arguments will be unquoted only once, but please do check the link for a more throughout explanation.
ExUnit provides a way to register a function as a test via register_test. It needs a context (__ENV__), what kind of test we want to register (:test, :property…), the name (which in this case corresponds to the endpoint URL), and a list of arguments.
Finally unquote will allow us to inject values in an AST. In this case, we’re creating a function named after the value of test_func_name with arguments stub, which we registered one line above.

Conclusion

Metaprogramming is a powerful tool, but with great power comes great responsibility. We must consider the tradeoff between the complexity a macro introduces and the benefit it gives us. I believe that that macro was simple enough to be in a test environment, and what’s better is it gave me a new method I can use in the future to test my code exhaustively. A few more examples that I’ve seen in the wild that could benefit from this method include:

• an event-based application where the order in which the events are processed cannot be predicted
• a patch\diff algorithm: in cases where applying the diffs in different order shouldn’t change the end result
• an expression calculator: combining different commutative zero-sum expressions shouldn’t change the result
• … more?

Originally published at prima.engineering on February 13, 2017.