Sit back and REST for a while [Chapter Two: Lift Off]
In the previous part of this series, we’ve covered the idea behind API testing in Fresha and initial steps for our Proof Of Concept. In this article, I’ll put more light on our approach and yes… there’s a plot twist!
Buckle up lads, we’re starting! 🚀 🔭
It quickly turned out that we might want to try 2 different approaches to the problem simultaneously in order to get more comprehensive feedback and have something to choose from. To make it even more diverse, the first solution will be a semi-free, out of the box and well known, good old friend Postman, while the second experiment a bare-bones Node.js custom framework built with open source libs and our own packages. Why did we choose exactly those two contestants and how they performed?
Please Mr. Postman 📯
I don’t think that this tool needs any introduction. It’s been with us for a while and proved that it’s reliable, potentially easy & quick to implement. It also has a lot of great features out of the box, is being actively developed, has a quite impressive community and is being used by a variety of bigger players like Microsoft or N26. It also has a buddy — Newman which is a cool CLI tool that enables you to smoothly implement Postman test suite runs on your CI with custom reports and a bunch of other awesome features.
Drawbacks? 📉 It might not be THAT scalable — I’ve heard from colleagues working in fintech companies with very robust microservices architecture and many internal/external API integrations that running and maintaining such huge test collections (because this is how we call a ‘test suite’ in Postman) is troublesome and prone to errors. In our case, we have around 600 different endpoints to cover and in the next couple of years we won’t need to scale this times 10 — so, not an issue.
The next thing to have in mind when you’re planning to use this particular tool is that — as I said before — it’s semi-free. What does it mean? Actually, it depends on how much hassle you can bear during the development process. Most of the core features (and more about to come) are free, however, some additional components like Monitoring which lets you set a repeating test suite to run those tests periodically on the specific endpoints and, for instance, assess the staging environment API health check.
You can pay and use that, but you don’t have to — there’s a smart workaround to this which actually is possible because of Postman and in some cases brings more value than the built-in one. It’s doable because of Newman — which, as mentioned is a CLI tool:
- You can write a script to import specific test collections in JSON files and provide additional data like environment variables, globals, data files including even CSV.
- What you can do next is to include an HTML report export in your script and re-upload of new env variables.
- Then just simply put that script on your own CI tool eg. Jenkins with installed Node and viola!
You have the custom monitoring and reporting feature. And there are many more examples of similar problems to be solved without a paid plan, but let’s stay focused on what was important for our project.
Furthermore, we wanted the tests to be written in BDD-like syntax because:
- it is what we do in our Cypress suites and we see the value of self-documenting tests
- it’s nice to have some idea on the testing scope, coverage in combination with TestRail and business-scoped test cases
Thankfully it’s not a problem, you can either use Postman’s sandbox built-in syntax like:
pm.test(‘Server response’, function () {
pm.response.to.have.status(200);
});Or you can do a little hack — which is importing an external tool like Mocha with Chai assertion library, then bundle it all up as a single javascript immediately invoked function expression — so-called IIFE. After that, you need to save that as a variable in postman sandbox (eg. globals). Here comes the hack — as postman does not validate values that you save up as variables, but instead just converts them into Strings you need to just simply paste eval() function in the Tests tab with reference to your BDD function from globals as an argument like:
eval(globals.yourBDDfunction);And that’s it! No, you can change your syntax to something like that:
describe(‘Common test’, () => {
it(‘Server response’, () => {
pm.response.to.have.status(200);
});
});Now you get this neat test result:
Well, that was unexpected 🙈
However, solving many problems along the way did not prepare us for what was coming next. In mid-February, there was another big Postman release, where some previously paid features were now free — but only up to some point. Whether it was the maximum number of requests or users — it had the limitations. The key point for us and many other companies is to protect their legacy — the code we write. To keep all our stuff someplace safe and easy to access — eg. on GitHub. At that point, it was clear to us that it’s going to be very challenging to keep reasonable version control over the Postman Collections which is, in fact, just a gigantic single JSON file.
After digging more into the topic it turned out that the developer workflow is complex: engineer working on test A needs to export postman_collection.json after he is finished to a local drive, then create the branch, commit changes and push it to the remote repo. Now another developer, before starting his work needs to pull that repo changes, create his own branch, do changes, commit, push, etc. — you know the drill. But, what happens when you try to merge 2 huge JSON files? Apparently in Postman, even a minor change, even one line of code will have an impact on the different parts of this JSON file. I don’t want to go deep into the technical details, but the point here is that it’s almost impossible to work like this and what is more — share the work over a couple of different people in a couple of different teams…
…It’s just not gonna work 💔.
The only option that makes sense is to go with a paid plan that includes Team Workspaces and some git-like features to pull, push, merge, commit the code among the teams, but it also makes you more dependent on the 3rd party tools and future changes to the planned scope, pricing levels, etc.
Another thing to consider is that it’s a sandbox that engineers need to learn how to use efficiently. It supports JavaScript, but still, you need to spend at least some time learning it and still, you won’t be able to use your favorite IDE, linters — unless you want to copy-paste the code over between Postman and your IDE all the time.
Last, but not least Postman comes with out of the box API documentation — both, as an interactive web page and additional complex, built-in APIs tool using, for example, Open API 3.0 format with YAML syntax to help you not only define & test, but also to develop. And of course, there’s much more of this kind of sugar in Postman and I’m pretty sure that if you find yourself in the position where this tool will fit your architectural and organizational needs you won’t regret using it!
The first attempt was not very successful for us, in the way, that it didn’t provide the final solution to the problem. However, it was still a valuable lesson and precious experience we’ll definitely use in the future. But… it’s not the end of this journey. We decided not to give up, but focus on the other tool that we already started building in the meantime. Are you interested in revealing the final stage of this mission? If so — stay tuned for the next and last article of this series! 📺
