Black-box Testing a Node.js Web API

15 min readMar 4, 2019

Black-box testing is a method of software testing that examines the functionality of an application without peering into its internal structures or workings. This method of test can be applied virtually to every level of software testing: unit, integration, system and acceptance. It is sometimes referred to as specification-based testing. — Wikipedia

In May of 2019 I read an excellent article from Yoni Goldberg titled “Node.js & JavaScript Testing Best Practices (2019)”. He made some great points in the article and it inspired me to improve my testing skills.

Update 2019–08–20: Yoni Goldberg has since created a GitHub repository titled JavaScript & Node.js Testing Best Practices.

A week later I was working on a side project and decided to adopt one of his best practices. Specifically point number 4. Stick to black-box testing: Test only public methods.

This article documents the process I went though to get to a true Web API black-box testing solution. It also includes specifics on how I achieved it.

This is not a tutorial. There is no repository on Github you can clone. It is more about the testing workflow and the parts required to get it working.

I am using the following technologies however the concepts will apply to all Node.js Web APIs:

Jest — a delightful JavaScript Testing Framework
express — Fast, unopinionated, minimalist web framework
MongoDB — A cross-platform document-oriented database
axios — Promise based HTTP client

Lost in the Woods

My Web API design breaks the request / response pipeline down into four layers being Routing, Controller, Store, and Database.

The Routing layer involves the standard express router configuration.
The Controller layer receives the request, response objects passed from the Routing layer and processes any query parameters. The Controller layer also wraps lower layer calls in a try/catch block.
The Store layer receives data requests from the Controllers and retrieves or manipulates the documents in the database. The request, response objects are not passed down to the Store layer.
The Database layer is simply MongoDB and a custom driver module.

I was working on the Store layer and decided to write integration tests that move up the stack and are closer to the black-box testing best practice. I remembered an NPM module that Yoni talked about in his post and decided to take a look at it. The module is called node-mock-http and he talks about it in point 15. Test your middlewares in isolation.

Here is the description of the node-mock-http module:

Mock ‘http’ objects for testing Express and Koa routing functions, but could be used for testing any Node.js web server applications that have code that requires mockups of the request and response objects.

If I used node-mock-http I would be moving the testing up to the Routing layer of my Web API. This sounded pretty good to me. Not quite black-box testing however only a thin layer of the API would not be tested.

As with any new NPM module you are evaluating, it is always a good idea to ask these quality questions:

Is it simple enough that I don’t need to add a dependency (roll your own)?
Is it maintained?
Is it popular?
Is it secure?
Is the module documented or does it have easy to read code (KISS)?
Is there a similar module that does the same thing and rates higher on the above questions?

Whilst researching the answers to the above questions for the node-mock-http module, I came across SuperTest.

Here is the description of the SuperTest module:

The motivation with this module is to provide a high-level abstraction for testing HTTP, while still allowing you to drop down to the lower-level API provided by superagent.

Note: SuperAgent is a HTTP module similar to axios.

If I used SuperTest I would be moving testing right to the top of the stack. This module was the answer to true black-box testing my Web API.

I shifted my module quality research to focus on SuperTest and settled on installing it and trying it out.

To quote a brilliant developer:

It was time to search for a tutorial using SuperTest. I love Jest and have been using the testing tool for some time. I didn’t want to drop Jest and just use the inbuilt assertions that are part of SuperTest. This lead me to a well written tutorial by Albert Gao titled “How to test Express.js with Jest and Supertest”.

The key takeaway I got from Albert’s article was to separate your web app object from the http server code or more specifically app.listen(3000).

I re-factored my server.js file by breaking it up into two files: server.js and app.js.

Note: the following code is not production ready. I need to evaluate the security modules prior to shipping.

Following is the app.js module file:

I’m not going to explain all the code above. If you want to understand it in more detail read the express documentation which is by far the best documentation I have seen on an open source project. When on the express documentation page simply CTLR+F in the browser and you will find what you are after.

It is worth highlighting a couple of important points. Firstly line 28 above is where the express app is instantiated, and secondly line 54 which exposes the app object.

I moved the server related code into a new file called server.js. Here is the server.js file content:

This file is used to launch the Web API and is the only module or layer of the stack that will not be under test. It has three main parts being variable assignment, exception handling, and server initialization. Following is a brief description of each section:

The variable assignment section simple imports the log, driver, app, and http port number values. This is where our app object is imported from the app.js file.
To ensure any unexpected errors are displayed on the console, there are two “if” statements adding exception handlers.
The server initialization section consists of an async function that connects my custom database driver to the MongoDB database and then enables the express app to listen on the server port. This is called at the end of the file with main().

Shameless plug: I use the perj module for logging.

I mostly work alone and therefore don’t have many developers to lean on. If I did, I might have discovered the following MongoDB package before now.

Datastores Galore

How do you do integration testing against a database? What are the options?

Here are the options I could think of prior to a new discovery:

Use the development MongoDB process and database for testing.
Use the development MongoDB process and test against a different database.
Install a new MongoDB process and database just for testing.

Note: I am ignoring containers for this article. The concepts are the same though.

Update 2020–03–06: I have moved away from using MongoDB thanks to inconsistencies. Read the article titled “MongoDB queries don’t always return all matching documents!”. Make sure you read the comments also. I am now using PostgreSQL for my side projects (why?). It is harder to work with but will give you more confidence in your data. If you are creating a large project, consider multiple data stores (SQL, NoSQL, NewSQL). Also take a look at YugabyteDB.

With MongoDB I thought these were the only options. I haven’t been using MongoDB for long because I was besotted by RethinkDB (which is mostly dead by the way). It was during my research into working with Jest and MongoDB that by chance I discovered this document on the Jest website: “Using with MongoDB”

The above document explains in detail how to run Jest integration tests against MongoDB. I got overly excited when I learned about an NPM module used in the examples called mongodb-memory-server. You little beauty. Lets add a new option to the three above:

Launch a new MongoDB in memory process for each test run.

I now had all the blocks I needed to do full isolated black-box integration testing.

But Why SuperTest?

I wont beat around the bush here. I had some trouble using SuperTest with Jest. I didn’t spend much time with it and I am sure that if I persisted I could have found the issues and fixed them. Instead I took a step back and looked at what SuperTest was doing.

Simply put, SuperTest is a HTTP client with some test assertions sprinkled on top. Why was I attempting to use SuperTest? I already had axios installed for a Node.js HTTP client. I was using Jest for test assertions. Ultimately I was just using SuperTest as a HTTP client only.

SuperTest is a great module and I suggest you give it a try if you think it is what you are after, however I didn’t need it.

I ditched SuperTest and decided to build my black-box integration tests using only axios as the request client.

The Final Solution

It took me a few hours to get all the pieces working together. When I finally got the testing process working it involved nine different files.

For anyone wanting to duplicate this process you will find all the code you need to repeat it below. Again, this is not a tutorial however with a little bit of gumption you will ascend to black-box testing.

1. Testing Workflow

Before discussing the files and configuration it is essential to understand the testing workflow.

Here are the steps I am using for black-box testing my Web API:

There is a lot going on in the above flow diagram so I’ll expand the steps below. Open this article in another browser window if you would like to reference the diagram whilst reading the steps:

Integration tests are initiated by running npm test.
The Jest setup process starts and reads the jest.config.js file.
The global-setup.js module is executed as referenced in the jest.config.js file. This module is rather simple for now and only executes db-setup.js.
db-setup.js launches an in-memory MongoDB instance, connects to the database, and adds required documents for authentication.
Now the Jest setup process spawns a child process for environment and test execution.
The jest.config.js file includes a testEnvironment key that points to the mongo-environment.js module. The Jest child process runs the setup() function from that module.
Jest scans the project looking for .test.js testing suite files.
Each test suite module includes a beforeAll() function which is executed before any tests.
From within beforeAll() the custom database driver is connected to the MongoDB in-memory instance.
Also from within beforeAll() the http-setup.js module is executed. This module launches the http server listener and authenticates against the user account added in db-setup.js. It sets the authentication cookie as an axios.defaults.
We finally get to running API tests. Any tests within the testing suite are carried out within the Jest child process.
At the end of the test suite file is the afterAll() function which is executed. This function closes the http server listener and the database driver is disconnected.
The final task of the Jest child process is to execute the teardown() function from the mongo-environment.js module.
Lastly the parent Jest process runs the gloabl-teardown.js module which stops the MongoDB in-memory instance.

Now that you have an understanding of the testing process, lets have a look at how it’s done.

2. Dependencies

Use npm to install the following dependencies prior to building the testing solution:

jest
jest-environment-node
mongodb-memory-server
axios

There are many more dependencies such as express, however they should already be installed for your application. Save them as devDependencies unless you wish to use axios in production.

3. NPM Test Script

You will need to create or update the test script in your package.json file.

Here is a simple example:

This is not the package.json file I am using. I generated it just to show the test script on line 8. Notice the environment variables I have set. You don’t need these. You could edit or delete them so that line 8 looks like this:

“test”: “jest --watch”

4. Jest Configuration

You will need to create a Jest config file in the root of your project if you don’t already have one.

Here is the one I am using:

The three files referenced in the Jest config will be discussed below. You will need to make sure your globalSetup, globalTeardown, and testEnvironment paths are correct.

5. Global Setup

The global-setup.js module is executed by the Jest setup process. Jest knows where to find the module file from reading the jest.config.js file.

For my use case the global-setup.js module is rather simple. It only executes the db-setup.js module.

Here is the contents of the global-setup.js file:

One thing to note about the Jest global setup process is that you can’t set global variable values for use in test suites. Tests are executed in a child process, not the Jest parent process. This is the reason for the testEnvironment Jest configuration which you will see below.

6. Database Setup

This is an important step in the setup process. The db-setup.js module gets executed by the global-setup.js module.

Before taking a look at the database setup step it is worth turning back to Yoni’s best practice article. In number 9. Avoid global test fixtures and seeds, add data per-test he recommends avoiding database initialization tasks. This is impossible in this example because each API call is required to be authenticated and authorized. The authentication is carried out against a user account document in the database and the authorization is based on policy documents. Due to this limitation the database needs some content for the tests to function.

Here is the contents of the db-setup.js file:

Line 10 creates a new in-memory MongoDB instance without starting it.

Line 14 exports the dbSetup function.

Line 19 to 22 creates a config object that holds the MongoDB connection URI. This URI is generated and is different for every test run.

Line 24 connects the custom database driver to the MongoDB in-memory instance.

Line 25 executes an internal module called initialize-db.js. This module is required as the Web API requires authentication and authorization to access it. The database needs to contain a test user account and access policy documents. I’m not going to show the contents of this module because it is specific to this application.

Line 28 writes the MongoDB configuration to a JSON file ,globalConfig.json, on the system. The global setup process executes in the context of the Jest setup process. To connect to the database in the test files, which are in a child process, the test suites will need to know the MongoDB connection URI. This config file persists the URI for later retrieval by the test suites.

Line 31 sets a global variable __MONGOD__ to the MongoDB instance object. This is used later in the global-teardown.js module to stop the process.

7. Test Environment

The Jest setup process is complete at this point. Now the mongo-environment.js module gets executed in a child process.

This module contains a setup(), teardown(), and runScript() function that all get called from Jest.

Here is the content of the mongo-environment.js file:

I haven’t changed this file at all. It is straight from the Jest example. Here is the link to the document for reference: “Using with MongoDB”

Line 18 and 19 load the saved globalConfig from the disk and set global variables. This module is executed in the Jest child process so the global variables are exposed to each test suite. These variables are used by the test suites to be able to connect to the in-memory database.

8. Integration Tests

Jest scans your project files for file names containing .test.js and others. Once it finds them it executes them in the child process.

Here is a contrived example of one test file:

Line 9 to 12 is the beforeAll() function which is called before any tests.

Line 10 connects the custom database driver to the in-memory MongoDB database. Note that it is using the global URI variable set from the mongo-environment.js module.

Line 11 calls the http-setup.js module. This is explained below in more detail however it enables the http server listener and authenticates axios against the API. A reference to the listener is kept for the afterAll() function.

Line 14 is the first integration test which is a PUT call against the API.

Line 24 is the winner. It is the purpose for this whole article. What we are doing here is acting like a standard http client and making a HTTP PUT request to the /products endpoint passing in the contrived product.

Line 25 is an example assertion. This simple example test suite should be expanded with many more assertions.

Line 29 is the second integration test which is a GET call against the API.

Line 31 is the second winner here. It is making a HTTP GET request to the /products endpoint to retrieve a list of products.

Line 32 and 33 are some simple test assertions. Again this should be expanded on.

Line 37 to 40 is the afterAll() function which is called after all the tests are complete.

Line 38 uses the http server listner to close the server.

Line 39 disconnects the custom database driver from the in-memory MondoDB instance.

9. HTTP Client Setup

When Jest executes a test suite the axios client is used to make calls to the Web API. That means we need the API or app in a listening state. The HTTP client, axios, also needs to be authenticated to be able to access the API.

All of this is achieved through the http-setup.js module.

Update 2019–08–20: Since writing this article I discovered I needed to test using application user accounts with different roles or even unauthenticated. I have changed the http-setup.js module to support a username and password argument. It also returns an instance of axios rather than changing the defaults.

Here is the content of the http-setup.js file:

Line 2 imports a shared axios client. It is important to realize this is a shared instance of axios. When we set axios.defaults it will apply to any module that imports axios within the Jest child process.

Line 3 imports our express application object

Line 6 exports the httpSetup() function which is the remainder of the file.

Line 8 starts the http server. A random high order TCP port is used.

Line 9 gets a reference to the generated http server port number.

Line 10 constructs the Web API endpoint URL.

Line 11 assigns a base URL to the shared axios client. This will be used as the http address for all future client requests unless it is overridden.

Line 13 is the authentication call to the Web API. This may be different in your application however I am using simple email and password authentication. A HAProxy will be used for SSL offloading in production.

Line 14 gets a reference to the valid authentication cookie.

Line 16 assigns the authentication cookie to the shared axios client HTTP header. As above, we are setting axios.defaults and it will apply to any axios client calls unless overridden.

Line 18 completes the function by returning the http server listener so the test suites can close the server on test completion.

10. Global Teardown

Finally after all test suite files have been run Jest executes the global-teardown.js module in the parent process. The global variables set in the global-setup.js module are available here.

All we are doing here is stopping the in-memory MongoDB instance.

Summary

There is a lot going on to achieve true black-box testing of a Node.js Web API. I believe the benefits outweigh the time required to set it up.

The major benefit I’m seeing right away is being able to build a new endpoint in the back-end of the application without needing to deal with the front-end. This helps me focus on one application design element, preventing multitasking from causing me to make more mistakes or perform tasks slowly.

The final solution section of this article is quite specific to Jest however the concepts will translate to any testing framework.

Thank you for taking the time to read this article. I hope you have learned something new from my discoveries.

Acknowledgements

I would like to thanks a few people and projects who made this work and article possible:

Yoni Goldberg, Albert Gao, and the many other people in this industry that make developing fun.

Node Weekly: A weekly email news letter I look forward to. Thanks Peter Cooper and staff. This is my main tool for finding articles like Yoni’s.

Medium: This is my first post on Medium. It seems like a great platform. Sub-lists would be helpful however all-in-all it is a great experience.

draw.io: What a brilliant tool. I used this web based drawing application to make the images in this article.

Jest: The guys working on Jest have and are doing a fantastic job. Keep it up.

axios: This project makes working with external HTTP endpoints a delight.

MongoDB: My expectations on document databases are high due to my work with RethinkDB. MongoDB comes close which is saying something. Well done.

mongodb-memory-server: Truly a brilliant target for application integration testing. Great work Pavel Chertorogov and the community.

Node.js: Finally Node.js and all the community involved in making a free open JavaScript platform that is a joy to work with.