How to integrate 3rd Party APIs without Fear (Part 2)

A journey between TDD, contract-tests and clean code

Welcome to the second part of ‘How to integrate third-party APIs without fear’. It’s recommended to read the previous part before proceeding with this one.

A quick recap: In the previous part we had introduced the Contract Testing concept. Our toy example was a Scala client library we wanted to write in order to send messages to Slack channels. We defined a list of requirements for that client and called it the ‘Contract’. This contract was an abstract class containing a few specifications (requirements; tests). It was an abstract class since it’s going to have two different implementations: one “real” and one “fake”.

You promised we’d see more code, right?

To satisfy the contract’s requirements, we decide that this is the trait (equivalent to Java’s interface, more or less) for our Slack Client:

The method returns a Try since we want to imply to the user of this class that bad things might happen in there. This way, we elegantly force the API consumer to consider failures rather than ignoring them — we keep her safe. We return a unit since posting a message is a side effect.

According to the first law of TDD — You are not allowed to write any production code unless it is to make a failing unit test pass — we’ll go ahead implement the first test:

slackClient is an abstract member that would be overridden by each one of the contract’s implementations. channelId is, for now, the ID of a real channel I had created for testing purposes.

Alas, when we try to run this test class we get something like:

org.specs2.control.UserException: cannot create an instance for class contract.slack.sync.SlackClientContract

That’s because it’s an abstract class. We must provide a concrete implementation if we wish to execute the tests. But which one will we choose? the Real or the Fake?

I prefer to begin with the real one. When a real test passes, I’m sure that everything is wired up correctly and the integration is perfect. Only then would I go ahead and write the fake as well.

A word regarding TokenProvider: we don’t want the API credentials to sit around in our code, rot, and god forbid — be committed to the VCS. So we’ll set it as a system property. Remember that this is the real API token we were given by Slack.

However, we have a compilation error in line 3. HttpSlackClient does not yet exist. Luckily enough, according to the second law of TDD — You are not allowed to write any more of a unit test than is sufficient to fail, and compilation failures are failures — we can now go ahead and write some actual production code.

Just to make sure everything is wired correctly, let’s run the test. As required, it fails with no less than my favorite exception. It ought to be meaning we’re doing something good (in Scala, three question marks are syntax sugar for throw new NotImplementedError).

java.lang.Exception: an implementation is missing

We’re set to go and can finally write some actual production code. I used scalaj-http to perform the HTTP request (because it’s a dead-simple library), and a standard Jackon’s object mapper to parse the response body. While doing so, try to remember the third and final law of TDD: You are not allowed to write any more production code than is sufficient to pass the one failing unit test.

Not only is the test green, but we can also see the message in the test channel we created in Slack.

Indeed, I had to play around until I got to this version of the code. It took me some time; I gradually built the HTTP request, debugged, and checked Slack’s response over and over again, and only after a few attempts have I managed to get it working.

But how does it differ from any other integration we had written in the past? That’s what we always do — play around with the code until we manage to get it all wired up correctly. Well, now we have a test. We can always go back and run it, and this way be sure the integration is still correct.

So far so code. But you mentioned that ‘real’ tests won’t even run within the CI build. What’s next?

Now we know for sure that our integration is correct. We have a ‘real’ test that gives us the strong validation we needed. The problem with this test is that it’s not isolated — it’s interacting with the real world. It might be flaky, slow, and probably shouldn’t run within our CI systems. Indeed, now it’s time to write the ‘fake’ tests class — which implements the same contract.

At this point, we stop, scratch our heads, and think. Our production code sends an HTTP request out to the world. That’s a side effect. How can we mimic the response of an HTTP request? We don’t have an object to mock & verify upon. If so, how can we write a test that would eventually pass?

For that, we need a Mock Server.

Luckily enough, this can be easily set up by using wix-http-testkit, which is an open-source library by Wix that helps exactly for these scenarios and is using akka-http under-the-hood.

Therefore, we’ll create a FakeSlackServer, which would resemble Slack within our test environment. The fake server would listen to the HTTP calls made by production code during tests execution, respond correspondingly, and satisfy our Slack client’s specifications. It will mimic Slack behavior just as much as we need it to — no less, no more.

Following is an initial implementation of FakeSlackServer, just enough to support the first requirement:

Pay special attention to the port our class receives as a member — that’s the port it would listen to. We will have to make sure that during the execution of the fake tests, our client would perform the HTTP calls to this exact port.

After having initiated the server on that port, we define a RequestHandler. RequestHandler is merely a partial function between HttpRequest and HttpResponse. We pattern match the request against the type of request we’d expect when posting a message to Slack, and return a valid response if indeed this is the case. For any other case, our server returns 500 by default.

Obviously, that’s a very thin implementation and there’s a lot more behavior we can resemble. We hadn’t fully validated the request body, nor did we verify the API token. but that’s just enough to make the first test green. The more tests we add, the more behavior we’d have to implement within FakeSlackServer.

Finally, we’re ready to write down the fake test class:

Take a look at how baseUrl is being accepted as a class member of HttpSlackClient. That’s because we wish to control where the HTTP calls would go to — real Slack or localhost. Executing the following class would result in a green test against the FakeSlackSever we had written.

It had cost us a few drops of sweat but we had finally completed a full TDD cycle. We have two green tests that cover the first requirement — one executes against ‘real’ Slack and the other against the ‘fake’ one. From now on, things will flow more smoothly.

Indeed, It’s much easier to continue now since we’re not afraid to change existing code — we have tests to verify there’s no regression. Also, the foundation of FakeSlackSever is already set up — we would just have to gradually extend it to support every new requirement we’re about to implement.

At any given point in time, we can go ahead and execute each of the concrete test classes we choose. If we need a strong validation we’ll run the real one. We’ll probably do that less, maybe after big changes, or before committing the code. When we don’t need as much assurance and want a quick feedback, we’ll execute the fake tests class. We’ll probably do that a lot.

For every new specification we write down in SlackClientContract, we must implement both concrete classes. We cannot escape or postpone it — we are forced to support the requirement for the two different environments. It keeps us safe and assured that each requirement we demand is correctly backed up in the client.

We’re done! check out the full code in GitHub to see the implementation for all of the requirements.

Summary

Third-party integrations can be weak spots within our systems. But they don’t have to. While reading this article, I hope you understood the importance of testing these integrations, and are eager to adopt the technique I had introduced.

You might argue that a lot of boiler platting is involved and that the process is too long. I would respond that after a few times you get the hang of it, and it flows much easier than at the beginning. The value is indeed higher than the cost.

Of course, you may say that you might not need integration tests at all. I’d say that there’s no single truth. Remember that each case is different, and try to judge every trade on its own merits.

Thank you for reading.