How to integrate 3rd Party APIs without Fear
A journey between TDD, contract-tests and clean code
Third-party APIs. How can we write our programs without them? It feels like we’ve reached a point in humanity where for almost every common problem, someone has already found a solution. So why not just use it? Why waste time over solving these problems, when we could be focusing 100% on our own business logic?
This is the first post out of a two parts article. If you have already read this one, feel free to jump to the second part.
Indeed, when writing an application nowadays we almost always find ourselves integrating with some external API. Whether it’s from a third-party, a micro-service someone from our company wrote, or even our own.
These kinds of integrations can be tricky. Tricky in such a way that we sometimes find ourselves tempted not to test them. That’s because writing integration tests for 3rd party API’s is not an easy task. ‘It’s an overhead’ we tell ourselves. Gloomy thoughts such as ‘Testing it will take too much time anyway’ often lurk inside our minds. The problem is that when we choose not to test, the next time we introduce a change within the integration code we’ll doom our future selves to deploy with fear, trembling in case we break that integration in production.
Learning the third-party code is hard. Integrating the third-party code is hard too. Doing both at the same time is doubly hard. What if we took a different approach? Instead of experimenting and trying out the new stuff in our production code, we could write some tests to explore our understanding of the third-party code.
— Robert C. Martin, Clean Code: A Handbook of Agile Software Craftsmanship
Here at Wix Engineering we respect our future selves, care deeply for resiliency, and embrace TDD (Test Driven Development) as a way of life. This means we have already adopted a methodology for these exact kinds of interactions. In this post, I’ll present one of our approaches to solving this problem, seasoned with a touch of my own ideology. Code examples are in Scala, which is a JVM, somewhat functional language. I chose it since that’s the language we mostly practice here at Wix Server guild, and also because, in my opinion, it’s one of the coolest programming languages out there.
What’s In It For Me? (Or: why should I read this?)
Once you’ve read this article, I hope you will:
- Fearlessly integrate with third-party APIs
- Sharpen your TDD skills
- Be a cleaner coder
- Earn some useful Scala tips & tricks
O.K., I’m convinced. What’s the first thing I need to know?
In this post, we’ll deep dive into a concept called Contract Testing. There are different ways and approaches to writing contract tests. Some are more complex, some simpler. I’ll focus on the methodology that I find to be the most convenient.
For me, contract testing is synonymous with expectation management. It’s like any other contract you sign — whether it’s a contract for an apartment, a job, or whatever. We commit to writing and sign over all of the expectations that both sides of the parties agree on.
The same thing goes for a third-party API — we have a contract with it, which describes the criteria we expect it to meet; the terms and conditions it must stand for, with respect to our needs.
A contract test could then be viewed as a list of requirements we expect this API to abide with. Reading the contract test gives us confidence that we know for sure how this API will behave during our application’s lifetime.
Sounds cool! But how exactly does it work?
The main idea is to eventually wind up with two implementations for one contract:
Real — code triggered by these tests would perform the HTTP calls to the real API server, out there in the external world, exactly as would happen in our production environment. These tests should not run within the CI system — they’re here to give strong confidence, but they don’t fit regular CI builds.
Fake — tests that would run against a fake server we write ourselves, or in some cases supplied by the SDK of the API provider (which is a good indicator we want to integrate with it). This so-called ‘server’ resembles and mimics the API behavior. HTTP calls would not go out to the external world, but rather to that fake server, started locally as a test collaborator.
Having these two kinds of implementations is, in my opinion, very important for the development velocity. ‘Real’ tests are slower to run, but significantly instill confidence in the correctness of our integration. We would probably want to run them after a big change, or before committing our code. They should not really be run as part of the CI build since they interact with the real world and might be flaky and/or too slow.
The ‘fake’ tests, on the other hand, would be faster to run and still give us a certain amount of confidence in the integration’s validation. Now, how much confidence they give us depends on how the fake server is written. The more it is accurate and resembles the real API, the more confidence we’ll have in these tests. We would probably execute these tests more often than the real ones. They’re fast, hermetic, and reproducible, giving us the confidence to run them as part of our regular CI builds, without bringing external instability.
The main benefit would be that a developer could easily juggle between executing the ‘real’ and the ‘fake’ tests while developing, without having to change any code or configuration whatsoever. If she needs a strong validation, she shall execute the ‘real’ tests. If she’ll be satisfied with less validation and wants quick feedback — she may execute the ‘fake’ tests. And all without changing a single line of code.
The concept is clear, but I think we need a ‘real world’ example.
In this toy example, let’s try and create an application that integrates with Slack. More specifically, we want to write some code that allows sending messages to channels. Consider the following background story: every time our system detects that some alert fires, it must send a message to our team’s urgent Slack channel.
For the sake of the example, let us zoom in to the integration point itself — that is the Client class we shall write for Slack. Let’s ignore the fact that there are a few pre-made Java clients for Slack out there for a moment — it’s a privilege we won’t, and don’t always have. And finally, to keep it really simple, the client is going to be synchronous.
The first thing we’d want to do is to study and play around with Slack’s Post Message endpoint. Then, we can write down the requirements we expect from our Slack Client. Those can be written as a list of specifications, or more specifically — tests. It’s a good hint as to why Specs2, a testing framework for Scala, is called the way it is.
To make things easier, I’ll sum up the essential information we need to know about that endpoint. I chose application/JSON as the request format.
POST https://slack.com/api/chat.postMessageBody:
{
"channel": {channel ID},
"text": {text}
}Headers:
Authorization: Bearer {token}
Content-Type: application/json
After studying the API, while considering how we need this client to behave within our application (remember that the client is an actor within a larger system that needs to interact with it), these might be the specifications we come up with. This will be our contract with the Slack:
Let’s break it down: the first requirement is pretty straight forward — our client simply needs to be able to post messages to channels. The second and the third one verify that it would provide its users (we, or whoever it is to use this client), the correct exception given they had provided a bad token/channel. We don’t want it to simply explode with some arbitrary null pointer/Parsing Exception, which would enforce an intensive debugging session in production.
At last, we have a contract. Now the sharper ones among you have probably noticed it’s an abstract class. That’s because this class merely defines the requirements, but knows nothing about the implementation details. there would be two concrete test classes to extend it — one for the ‘real’ tests, and the other for the ‘fake’ ones. The key difference between them would be the environment setup — and not the tests themselves.
The first concrete class (‘real’) will initiate the SUT (system under test) so that it would perform HTTP calls directly to Slack, exactly as would happen in production. When we execute this class, we expect to see the output of our production code in real life with our own eyes— In this case, see a new message in the appropriate Slack channel.
The second concrete class (‘fake’) would start up a fake server before tests begin to run, and configure our client to make the calls to that server, rather than to Slack itself. Production code triggered by this class would interact with that server rather than Slack. We won’t see any message being sent anywhere, but if we have enough trust in that server, we shall be satisfied when we see these tests green.
As promised at the beginning — one contract, two implementations.
Thanks for reading!
That was the first part of my blog-post ‘How to integrate third-party APIs without fear’. In the next part, we’ll continue TDD’ing our way through the example, and see more code rather than words. It would be more of a technical, Scala-oriented post, and a less conceptual one. Stay tuned!
