Black-box testing is a method of software testing that examines the functionality of an application without peering into its internal structures or workings. This method of test can be applied virtually to every level of software testing: unit, integration, system and acceptance. It is sometimes referred to as specification-based testing. — Wikipedia
A week later I was working on a side project and decided to adopt one of his best practices. Specifically point number 4. Stick to black-box testing: Test only public methods.
This article documents the process I went though to get to a true Web API black-box testing solution. It also includes specifics on how I achieved it.
This is not a tutorial. There is no repository on Github you can clone. It is more about the testing workflow and the parts required to get it working.
I am using the following technologies however the concepts will apply to all Node.js Web APIs:
- express — Fast, unopinionated, minimalist web framework
- MongoDB — A cross-platform document-oriented database
- axios — Promise based HTTP client
Table of Contents
Lost in the Woods
My Web API design breaks the request / response pipeline down into four layers being Routing, Controller, Store, and Database.
- The Routing layer involves the standard express router configuration.
- The Controller layer receives the
request, responseobjects passed from the Routing layer and processes any query parameters. The Controller layer also wraps lower layer calls in a
- The Store layer receives data requests from the Controllers and retrieves or manipulates the documents in the database. The
request, responseobjects are not passed down to the Store layer.
- The Database layer is simply MongoDB and a custom driver module.
I was working on the Store layer and decided to write integration tests that move up the stack and are closer to the black-box testing best practice. I remembered an NPM module that Yoni talked about in his post and decided to take a look at it. The module is called node-mock-http and he talks about it in point 15. Test your middlewares in isolation.
Here is the description of the node-mock-http module:
Mock ‘http’ objects for testing Express and Koa routing functions, but could be used for testing any Node.js web server applications that have code that requires mockups of the request and response objects.
If I used node-mock-http I would be moving the testing up to the Routing layer of my Web API. This sounded pretty good to me. Not quite black-box testing however only a thin layer of the API would not be tested.
As with any new NPM module you are evaluating, it is always a good idea to ask these quality questions:
- Is it simple enough that I don’t need to add a dependency (roll your own)?
- Is it maintained?
- Is it popular?
- Is it secure?
- Is the module documented or does it have easy to read code (KISS)?
- Is there a similar module that does the same thing and rates higher on the above questions?
Whilst researching the answers to the above questions for the node-mock-http module, I came across SuperTest.
Here is the description of the SuperTest module:
The motivation with this module is to provide a high-level abstraction for testing HTTP, while still allowing you to drop down to the lower-level API provided by superagent.
Note: SuperAgent is a HTTP module similar to axios.
If I used SuperTest I would be moving testing right to the top of the stack. This module was the answer to true black-box testing my Web API.
I shifted my module quality research to focus on SuperTest and settled on installing it and trying it out.
To quote a brilliant developer:
It was time to search for a tutorial using SuperTest. I love Jest and have been using the testing tool for some time. I didn’t want to drop Jest and just use the inbuilt assertions that are part of SuperTest. This lead me to a well written tutorial by Albert Gao titled “How to test Express.js with Jest and Supertest”.
The key takeaway I got from Albert’s article was to separate your web app object from the http server code or more specifically
I re-factored my
server.js file by breaking it up into two files:
Note: the following code is not production ready. I need to evaluate the security modules prior to shipping.
Following is the
app.js module file:
I’m not going to explain all the code above. If you want to understand it in more detail read the express documentation which is by far the best documentation I have seen on an open source project. When on the express documentation page simply CTLR+F in the browser and you will find what you are after.
It is worth highlighting a couple of important points. Firstly line 28 above is where the express app is instantiated, and secondly line 54 which exposes the app object.
I moved the server related code into a new file called
server.js. Here is the
server.js file content:
This file is used to launch the Web API and is the only module or layer of the stack that will not be under test. It has three main parts being variable assignment, exception handling, and server initialization. Following is a brief description of each section:
- The variable assignment section simple imports the log, driver, app, and http port number values. This is where our app object is imported from the
- To ensure any unexpected errors are displayed on the console, there are two “if” statements adding exception handlers.
- The server initialization section consists of an async function that connects my custom database driver to the MongoDB database and then enables the express app to listen on the server port. This is called at the end of the file with
Shameless plug: I use the perj module for logging.
I mostly work alone and therefore don’t have many developers to lean on. If I did, I might have discovered the following MongoDB package before now.
How do you do integration testing against a database? What are the options?
Here are the options I could think of prior to a new discovery:
- Use the development MongoDB process and database for testing.
- Use the development MongoDB process and test against a different database.
- Install a new MongoDB process and database just for testing.
Note: I am ignoring containers for this article. The concepts are the same though.
Update 2020–03–06: I have moved away from using MongoDB thanks to inconsistencies. Read the article titled “MongoDB queries don’t always return all matching documents!”. Make sure you read the comments also. I am now using PostgreSQL for my side projects (why?). It is harder to work with but will give you more confidence in your data. If you are creating a large project, consider multiple data stores (SQL, NoSQL, NewSQL). Also take a look at YugabyteDB.
With MongoDB I thought these were the only options. I haven’t been using MongoDB for long because I was besotted by RethinkDB (which is mostly dead by the way). It was during my research into working with Jest and MongoDB that by chance I discovered this document on the Jest website: “Using with MongoDB”
The above document explains in detail how to run Jest integration tests against MongoDB. I got overly excited when I learned about an NPM module used in the examples called mongodb-memory-server. You little beauty. Lets add a new option to the three above:
- Launch a new MongoDB in memory process for each test run.
I now had all the blocks I needed to do full isolated black-box integration testing.
But Why SuperTest?
I wont beat around the bush here. I had some trouble using SuperTest with Jest. I didn’t spend much time with it and I am sure that if I persisted I could have found the issues and fixed them. Instead I took a step back and looked at what SuperTest was doing.
Simply put, SuperTest is a HTTP client with some test assertions sprinkled on top. Why was I attempting to use SuperTest? I already had axios installed for a Node.js HTTP client. I was using Jest for test assertions. Ultimately I was just using SuperTest as a HTTP client only.
SuperTest is a great module and I suggest you give it a try if you think it is what you are after, however I didn’t need it.
I ditched SuperTest and decided to build my black-box integration tests using only axios as the request client.
The Final Solution
It took me a few hours to get all the pieces working together. When I finally got the testing process working it involved nine different files.
For anyone wanting to duplicate this process you will find all the code you need to repeat it below. Again, this is not a tutorial however with a little bit of gumption you will ascend to black-box testing.
1. Testing Workflow
Before discussing the files and configuration it is essential to understand the testing workflow.
Here are the steps I am using for black-box testing my Web API:
There is a lot going on in the above flow diagram so I’ll expand the steps below. Open this article in another browser window if you would like to reference the diagram whilst reading the steps:
- Integration tests are initiated by running
- The Jest setup process starts and reads the
global-setup.jsmodule is executed as referenced in the
jest.config.jsfile. This module is rather simple for now and only executes
db-setup.jslaunches an in-memory MongoDB instance, connects to the database, and adds required documents for authentication.
- Now the Jest setup process spawns a child process for environment and test execution.
jest.config.jsfile includes a
testEnvironmentkey that points to the
mongo-environment.jsmodule. The Jest child process runs the
setup()function from that module.
- Jest scans the project looking for
.test.jstesting suite files.
- Each test suite module includes a
beforeAll()function which is executed before any tests.
- From within
beforeAll()the custom database driver is connected to the MongoDB in-memory instance.
- Also from within
http-setup.jsmodule is executed. This module launches the http server listener and authenticates against the user account added in
db-setup.js. It sets the authentication cookie as an
- We finally get to running API tests. Any tests within the testing suite are carried out within the Jest child process.
- At the end of the test suite file is the
afterAll()function which is executed. This function closes the http server listener and the database driver is disconnected.
- The final task of the Jest child process is to execute the
teardown()function from the
- Lastly the parent Jest process runs the
gloabl-teardown.jsmodule which stops the MongoDB in-memory instance.
Now that you have an understanding of the testing process, lets have a look at how it’s done.
npm to install the following dependencies prior to building the testing solution:
There are many more dependencies such as express, however they should already be installed for your application. Save them as
devDependencies unless you wish to use axios in production.
3. NPM Test Script
You will need to create or update the test script in your
Here is a simple example:
This is not the
package.json file I am using. I generated it just to show the test script on line 8. Notice the environment variables I have set. You don’t need these. You could edit or delete them so that line 8 looks like this:
“test”: “jest --watch”
4. Jest Configuration
You will need to create a Jest config file in the root of your project if you don’t already have one.
Here is the one I am using:
The three files referenced in the Jest config will be discussed below. You will need to make sure your
testEnvironment paths are correct.
5. Global Setup
global-setup.js module is executed by the Jest setup process. Jest knows where to find the module file from reading the
For my use case the
global-setup.js module is rather simple. It only executes the
Here is the contents of the
One thing to note about the Jest global setup process is that you can’t set global variable values for use in test suites. Tests are executed in a child process, not the Jest parent process. This is the reason for the
testEnvironment Jest configuration which you will see below.
6. Database Setup
This is an important step in the setup process. The
db-setup.js module gets executed by the
Before taking a look at the database setup step it is worth turning back to Yoni’s best practice article. In number 9. Avoid global test fixtures and seeds, add data per-test he recommends avoiding database initialization tasks. This is impossible in this example because each API call is required to be authenticated and authorized. The authentication is carried out against a user account document in the database and the authorization is based on policy documents. Due to this limitation the database needs some content for the tests to function.
Here is the contents of the
Line 10 creates a new in-memory MongoDB instance without starting it.
Line 14 exports the
Line 19 to 22 creates a config object that holds the MongoDB connection URI. This URI is generated and is different for every test run.
Line 24 connects the custom database driver to the MongoDB in-memory instance.
Line 25 executes an internal module called
initialize-db.js. This module is required as the Web API requires authentication and authorization to access it. The database needs to contain a test user account and access policy documents. I’m not going to show the contents of this module because it is specific to this application.
Line 28 writes the MongoDB configuration to a JSON file ,
globalConfig.json, on the system. The global setup process executes in the context of the Jest setup process. To connect to the database in the test files, which are in a child process, the test suites will need to know the MongoDB connection URI. This config file persists the URI for later retrieval by the test suites.
Line 31 sets a global variable
__MONGOD__ to the MongoDB instance object. This is used later in the
global-teardown.js module to stop the process.
7. Test Environment
The Jest setup process is complete at this point. Now the
mongo-environment.js module gets executed in a child process.
This module contains a
runScript() function that all get called from Jest.
Here is the content of the
I haven’t changed this file at all. It is straight from the Jest example. Here is the link to the document for reference: “Using with MongoDB”
Line 18 and 19 load the saved
globalConfig from the disk and set global variables. This module is executed in the Jest child process so the global variables are exposed to each test suite. These variables are used by the test suites to be able to connect to the in-memory database.
8. Integration Tests
Jest scans your project files for file names containing
.test.js and others. Once it finds them it executes them in the child process.
Here is a contrived example of one test file:
Line 9 to 12 is the
beforeAll() function which is called before any tests.
Line 10 connects the custom database driver to the in-memory MongoDB database. Note that it is using the global URI variable set from the
Line 11 calls the
http-setup.js module. This is explained below in more detail however it enables the http server listener and authenticates axios against the API. A reference to the listener is kept for the
Line 14 is the first integration test which is a PUT call against the API.
Line 24 is the winner. It is the purpose for this whole article. What we are doing here is acting like a standard http client and making a HTTP PUT request to the
/products endpoint passing in the contrived product.
Line 25 is an example assertion. This simple example test suite should be expanded with many more assertions.
Line 29 is the second integration test which is a GET call against the API.
Line 31 is the second winner here. It is making a HTTP GET request to the
/products endpoint to retrieve a list of products.
Line 32 and 33 are some simple test assertions. Again this should be expanded on.
Line 37 to 40 is the
afterAll() function which is called after all the tests are complete.
Line 38 uses the http server listner to close the server.
Line 39 disconnects the custom database driver from the in-memory MondoDB instance.
9. HTTP Client Setup
When Jest executes a test suite the axios client is used to make calls to the Web API. That means we need the API or app in a listening state. The HTTP client, axios, also needs to be authenticated to be able to access the API.
All of this is achieved through the
Update 2019–08–20: Since writing this article I discovered I needed to test using application user accounts with different roles or even unauthenticated. I have changed the
http-setup.js module to support a
password argument. It also returns an instance of
axios rather than changing the defaults.
Here is the content of the
Line 2 imports a shared axios client. It is important to realize this is a shared instance of axios. When we set
axios.defaults it will apply to any module that imports axios within the Jest child process.
Line 3 imports our express application object
Line 6 exports the
httpSetup() function which is the remainder of the file.
Line 8 starts the http server. A random high order TCP port is used.
Line 9 gets a reference to the generated http server port number.
Line 10 constructs the Web API endpoint URL.
Line 11 assigns a base URL to the shared axios client. This will be used as the http address for all future client requests unless it is overridden.
Line 13 is the authentication call to the Web API. This may be different in your application however I am using simple email and password authentication. A HAProxy will be used for SSL offloading in production.
Line 14 gets a reference to the valid authentication cookie.
Line 16 assigns the authentication cookie to the shared axios client HTTP header. As above, we are setting
axios.defaults and it will apply to any axios client calls unless overridden.
Line 18 completes the function by returning the http server listener so the test suites can close the server on test completion.
10. Global Teardown
Finally after all test suite files have been run Jest executes the
global-teardown.js module in the parent process. The global variables set in the
global-setup.js module are available here.
All we are doing here is stopping the in-memory MongoDB instance.
There is a lot going on to achieve true black-box testing of a Node.js Web API. I believe the benefits outweigh the time required to set it up.
The major benefit I’m seeing right away is being able to build a new endpoint in the back-end of the application without needing to deal with the front-end. This helps me focus on one application design element, preventing multitasking from causing me to make more mistakes or perform tasks slowly.
The final solution section of this article is quite specific to Jest however the concepts will translate to any testing framework.
Thank you for taking the time to read this article. I hope you have learned something new from my discoveries.
I would like to thanks a few people and projects who made this work and article possible:
Medium: This is my first post on Medium. It seems like a great platform. Sub-lists would be helpful however all-in-all it is a great experience.
draw.io: What a brilliant tool. I used this web based drawing application to make the images in this article.
Jest: The guys working on Jest have and are doing a fantastic job. Keep it up.
axios: This project makes working with external HTTP endpoints a delight.
MongoDB: My expectations on document databases are high due to my work with RethinkDB. MongoDB comes close which is saying something. Well done.