Automating Test Automation

Image by Geralt — https://pixabay.com/en/users/geralt-9301/

Abstract

In this article I would like to offer an unorthodox approach to software testing (API clients in particular). The article will make the case for starting with (basic) integration testing and then reusing these same (mocked) test cases for unit testing.

Background

Over the past two weeks I had the opportunity to contribute to the node-alexa-smapi open source project. As you can see, the project code itself is pretty straightforward and my code contribution to add Alexa SMAPI v1 support was completed in the first few days.

Here is the catch: my contribution to the code was done according to the documented API… Now what years of experience have taught me is that what is documented is not necessarily how it is working right now. Does that sound familiar to you?

So right after finishing the code, I immediately started using Mocha, Chai, and chai-as-promised to write test cases so I could:

• Make sure the code was working for the live (rather than documented) API.
• Double check that the way I understood the API should work was actually the way it worked.
• Improve the code quality by adding test cases.

Unit Testing or Integration Testing?

So a week later code and test cases were ready for the pull request. I was feeling pretty good as our first code coverage report came in at 82% coverage. More importantly, the test cases allowed me to get a better understanding of SMAPI and to implement a few code improvements.

At this point the library author asked me about adding unit test cases. And right he was! I was so focused on checking how SMAPI actually worked that all my test cases so far were (basic) integration tests.

Computer Science concept: so, if I am achieving 82% test coverage why do I care if I am running unit tests or integration tests?

… (a unit test) should not cross such process/network boundaries because this can introduce unacceptable performance problems to the unit test-suite. (Wikipedia)

And by running integration tests the Travis CI build was taking 47 minutes (due to throttling, retries, etc.) to complete and occasionally it would fail (e.g. due to timeout).

Mocked Up Integration Test = Unit Test?

So here was my new challenge:

• “Capture the JSON responses of all the requests and mock the API calls for unit testing” (Tejas Shah)
• Make this test automated (i.e. generating the JSON responses regularly).

So, I am now introduced to axios-mock-adapter and started figuring out how to best tackle the task at hand…

Turns out axios-mock-adapter made life pretty easy! As in a sleight of hand I was able to reuse all the existing test suite with minimal changes to both automatically capture the JSON responses and run unit tests!!!

Magicians Never Reveal Their Secrets…

Then it is a good thing “open sourcers” do! Here they are…

The test suite has now four operation modes (test suite run times might vary, these were taken from my computer):

• Unit mode: will use the automatically captured JSON responses to execute the test suite. In this mode the test suite completes in about 112 milliseconds and the Travis CI build completes in less than 4 minutes!
• Integration mode: will connect live to Alexa SMAPI servers to execute the test suite (excluding Alexa certification tests). In this mode the test suite completes in about 2 minutes.
• Certification mode: will connect live to Alexa SMAPI servers to execute the test suite (including Alexa certification tests). In this mode the test suite completes in about 7 minutes.
• Capture mode: same as certification mode plus it will automatically capture the JSON responses.

When no stored JSON responses exist, the testing workflow works like this:

1. Run the test suite in integration mode to check the API works as expected.
2. Run the test suite in capture mode before committing code and source control the JSON responses along with the code.
3. CI/CD pipelines and future (regression) tests should run the test suite in unit mode.

And here is a summary insight into the code changes required to implement the four modes:

• Mock adapter: boiler plate code to load the captured JSON responses. Please note that a few JSON responses were added manually to avoid exposing sensitive data (access tokens in our case).
• Response capture: boiler plate code to capture and save the JSON responses. Few details worth mentioning: only meaningful headers (etag & location in SMAPI’s case) were stored. In order to improve maintainability and security, any data which changes between runs was “sanitized”.
• Test suite instrumentation: added helper functions (e.g. shouldMock, shouldCapture), test type in order to change the test suite behavior accordingly in different operation modes, used “after” clauses to sanitize & persist JSON responses accordingly. Finally while all success (2XX HTTP status) responses were captured (using axios response interceptors), only a few error responses were kept (the ones required by error scenarios — see variable expectedErrorKeywords).

Key Take Aways

As this approach allowed us in a few days to add unit tests and reach 98% code coverage with little changes to the initial (basic) integration tests, I pondered about lessons learned and here are my key take aways:

• As you right code to check out and get acquainted with existing functionality, start writing it within a testing framework (Mocha/Chai in our case). These test cases can easily become your (basic) integration tests and can also serve you as regression tests in the future as the server side API matures.
• Be sure to pick libraries (axios and axios-mock-adapter in our case) which will allow you to easily and non-intrusively (from the point of view of writing test cases) mockup requests/responses.
• Be obsessive about automation. While it would have been “faster” to manually copy/paste the JSON responses, it makes it harder to maintain the code (and test suite) in the long run as the server API goes through changes.

I hope to have stirred up some thoughts with this unorthodox testing approach! Is it worth capturing this approach into a library of its own? Feel free to share your experience and point of view.