Testing External Web Requests in Elixir? Roll Your Own Mock Server

Pictured: your very own hand-rolled mock server

What should you do when testing Elixir code that makes web requests to an external API? We don’t want to let our code make those requests during a test run — we’ll slow down our tests and potentially use up API rate limits. We could mock such requests with the help of a mocking library or use recorded requests with the help of recording library. We’ll show you why you should avoid these approaches and instead roll your own mock server for your test environment.

Don’t: Use Recorded Requests

While there are a number of web request recording libraries in Elixir, there are *also* a number of drawbacks to using them:

• You’re adding a dependency to your test suite.
• You’re bloating your file tree with potentially lots and lots of “cassette” files.
• It can be onerous to record a “first cassette” for each request you need to “playback.”
• It can be onerous to re-record cassettes if your code changes the manner in which it hits an API endpoint or if the API itself changes.
• Imitating sad paths can be tricky, since you have to create real failed requests for every type of anticipated failure.

Don’t: Mock Web Requests

You’re convinced that you *don’t* want to use recorded requests (great!). So how about mocking web requests instead?

Let’s say you’re building an application that talks to the GitHub API. You have a module, GithubClient, that uses HTTPoison to make web requests to that API. You could mock these web requests by mocking the function calls made against HTTPoison. Let’s say we are trying to test the following code:

Using the Elixir Mock library, your test would look something like this:

So, what’s so wrong here? Well, first of all, we’ve now coupled our tests directly to the HTTPoison dependency. If we choose to switch HTTP clients at a later time, all of our tests would fail, even if our app still behaves correctly. Not fun.

Second, this makes for some really repetitious tests. Any tests that run code that hit the same API endpoints will require us to mock the requests all over again. Even these two tests for our happy and sad create_rep paths feel repetitious and clunky.

The repetitious mocking code that now bloats our tests make them even harder to read. Instead of letting contexts and it descriptors speak for themselves, anyone reading out tests has to parse the meaning of lots and lots of inline mocking code.

Lastly, this approach not only makes our lives more difficult but also represents a misuse of mocks. Jose Valim’s article on this topic describes it best:

Mocks are simulated entities that mimic the behavior of real entities in controlled ways…I always consider “mock” to be a noun, never a verb.

By “mocking” (verb!) our API interactions, we make an implementation detail (the fact that we are using `HTTPoison`) a first class citizen in our test environment, with the ability to break all of our tests, even when our app behaves successfully.

Instead of misusing mocks in this manner, let’s build our very own mock (noun!) server and run it in our test environment.

Do: Build Your Own Mock Server

We won’t be mocking calls to HTTPoison functions. We’ll let HTTPoison make real web requests. BUT! Instead of sending those requests to the GitHub API, we’ll make the API URL configurable based on environment, and tell HTTPoison to hit our own internally served “test endpoint” in the test environment.

We’ll build a simple HTTP server in our Elixir app, using Cowboy and Plug. Then, we’ll define a controller that knows how to respond to the requests that HTTPoison will send in the course of the test run of our GithubClient code. Lastly, we’ll configure our app to run our server in the test environment only.

Let’s get started!

Building the Mock HTTP Server with Cowboy and Plug

Our Github Client app is not a Phoenix application. So if we want to run a server, we need to build that server ourselves. Luckily, Cowboy and Plug make it pretty easy to set up a simple HTTP server in our Elixir app. Cowboy is a simple HTTP server for Erlang and Plug provides us with a connection adapter for that web server.

First things first, we’ll add the PlugCowboy library to our app’s dependencies in the mix.exs file.

We’ll tell our application to run Cowboy and Plug in the test environment only:

Next up, we’ll define our GithubClient.MockServer module. We want our server to do a few things for us:

• Start up in a supervision tree when the application starts (in the test env only)
• Handle and respond to incoming web requests.

In order to make sure we can start up our test server as part of our application’s supervision tree, it will need to implement the GenServer API’s init and start_link functions. Luckily for us, we’ll get that behavior by using the Plug.Router module.

Now, we can tell our application to start up and supervise the Cowboy web server when the app starts up.

We’ll do so with the Plug.Cowboy.child_spec/1 function.

This function expects three options:

• :scheme - HTTP or HTTPS as an atom (:http, :https)
• :plug - The plug module to be used as the interface for the web server. You can specify a module name, like MyPlug, or a tuple of the module name and options {MyPlug, plug_opts}, where plug_opts gets passed to your plug modules init/1 function.
• :options - The server options. Should include the port number on which you want your server listening for requests.

Now that our server knows how to start up, let’s build out the GithubApiMockServer that is the interface for our web requests. We’ll give it some routes and teach it how to respond to certain web requests.

Building the Github API Mock Controller

Our GithubApiMockServer module is the interface for our web server. It needs to know how to route requests and respond to the specific requests that our GithubClient will send in the course of a test run. The mock server will act as a stand in for the GitHub API, expect to receive all of the web requests that we would send to GitHub and respond accordingly.

In order for our controller to route web requests, we need to tell it to use Plug.Router This provides us the routing macros we need to match and respond to web requests.

Since our controller will be receiving JSON payloads (just like the GitHub API!), we’ll also tell it to run requests through the Plug.Parsers plug. This will parse the request body for us.

Now we’re ready to add our routes!

Defining Routes For Our Mock

Eventually, we’ll need to add routes that know how to handle the happy and sad paths for any web requests sent in the course of a test run. For now, we’ll revisit our earlier test example, which runs code that hits the `POST /repos` GitHub API endpoint:

Here we’ve defined a route POST /repos that uses a case statement to introspect on some params and send a happy or sad response.

Now that our mock server’s interface is defined to handle this request, let’s refactor our tests.

Cleaner Tests with Our Mock Server

First, a quick refresher on the code we’re testing. The GithubClient.create_repo function does two things:

• Make the POST request to the /repos endpoint
• Handle the response to return a GithubRepo struct or a GithubError struct.

Our code looks something like this:

We want to test that, when we successfully create a repo via the GitHub API, the function returns a GithubRepo struct. When we don’t successfully create a repo via the GitHub API, we return a GithubError struct. Instead of defining complicated function mocks inside our tests, we’ll write our nice clean tests with no awareness of any mocks.

In order for our tests to use our mock server, we need to make one simple change: tell the GithubClient module to send requests to our internally hosted endpoint in the test environment, instead of to the GitHub API.

To do that, we’ll *stop* hard-coding the value of the @base_url module attribute and instead make it an environment-specific application variable:

Now we can write tests that are totally agnostic of any mocking:

Ta-da!

Conclusion

By writing our own mock server, we are able write tests that illustrate and test the contract or interface between our own code and the external GitHub API. We are testing that our code behaves as expected, given an expected response. This is different from *mocking* an HTTP client object, which requires us to simulate the behavior of an object that is not a necessary part of our app’s successful communication with the API.

By remember that by creating *mocks* (noun!) instead of *mocking* (verb!), we end up with clean, readable tests that don’t rely on implementation details to pass. So next time you’re faced with testing code that makes external web requests, remember that a simple hand-rolled mock server is now part of your tool belt.

Want to work on a mission-driven team that loves hand-rolled mock servers and trust-worthy tests? We’re hiring!

To learn more about Flatiron School, visit the website, follow us on Facebook and Twitter, and visit us at upcoming events near you.

Flatiron School is a proud member of the WeWork family. Check out our sister technology blogs WeWork Technology and Making Meetup.