End-to-End Testing in the Development Environment
Author: Nakul Garg
Motive
The development environment of a software system is typically the Wild West for a technology-oriented company. But at Latch, we think it’s necessary to formally test this chaos. We believe there’s valuable insight gathered during this process—helping engineers discover bugs earlier on and reducing the chances of one ever reaching production. By running a test suite after every merge, developers can know within minutes whether their recently pushed code has issues. More specifically, routine integration tests can catch system regressions by simulating the real world environment. Here, we outline our testing framework, which can apply to any development environment, with the intention of being lightweight and routinely-run.
Technical Overview
Due to limited resources, we prioritized ease and speed of development in our test environment. These values naturally led us to develop in Python, using behavior-driven development (or BDD) called Behave. This technique allows tests to be defined in plain English in feature files and is automatically mapped to Python—allowing people of various technical backgrounds to methodically define their tests that engineers can then easily digest and translate into code. We’ll go further into detail about Behave later in this article, and we also provide our recommended directory structure as a great starting point.
We also prefer Behave because it seamlessly integrates with Allure test reporting, which then conveniently integrates with Jenkins, a popular open source automation server. This whole process takes only minutes to set up and run, resulting in an Allure report link that can be shared via Slack, email, and other channels. These three frameworks quickly work together to provide routine reports about the state of features in development.
Behave
To get started, we’ll provide basic instruction to quickly put together a test suite. To learn more about the Behave framework, please see the actual documentation.
The Environment File (environment.py)
The environment file where developers using Behave can define a function that runs before every test. This is useful to populate Behave’s “context” concept. The “context” is a variable in which steps can add data in order to pass information. The “before_all” function is a great place to define global variables about your system that will be useful to have available across tests.
Feature Files
Feature files are a plain English definition of the test, but they can be set up so that they organize when and which code is run. Feature files are defined with a short description.
Next, it’s important to declare a “scenario” or test procedure. We define a scenario outline with a test name. Features can have any number of scenarios, and each scenario can be deployed when the feature is run. The following lines starting with “Given,” “When,” or “Then” represent steps in the test. “Given” steps set up base information,“When” steps utilize information in some way, and “Then” steps assess the expected state.
These are the components that map directly to a Python step implementation. As a result, we can conveniently reuse these steps by using the same language since Behave will automatically match it back to implementation steps. There’s no limit to the number of steps in a scenario.
Information can be passed to a step through a <parameter>. The first row of the Examples section defines the parameters, and each of the following rows defines sample values. For example, the first run of the scenario above tests that user1@place.com with user1password gets thing1 and thing2 when logging in. The second run tests that user2@place.com with user2password has no things when logging in. If the expected information is large, then parameters can represent a file path which can be opened from the Python implementation. Lastly, we can tag scenarios and run a test set based off of their tags.
Python Step Files
Python step files are where the actual implementation lives. Through Behave’s function tags, the steps that we defined in our Scenarios can be linked easily to Python implementations. The actual function name does not matter because all linking will come from tag. The “@given” tag is used above, but the format is the same for @when and @then. Parameters from the feature step must have {} around them rather than the <> in the Feature.
Next, it’s necessary to define the parameters as actual function inputs with matching names, and the information is readily available. Then, all we have to do is actually implement the step. For example, the step shown above hits an endpoint to log in and stores the response in the context for the next step to deal with.
The Package File (package.json)
In order to make running tests from the command line more convenient, we threw together a quick package.json file. From the example above, we can put “npm run dev” into the command line which will run every test with the dev tag. The Allure Formatter is discussed later in this article.
Recommended Directory Structure
Behave mandates a rigid directory setup with some peculiar rules — the most unexpected being that a “steps” directory must contain all Python step files without any nested directories. This rule can be an eyesore for any system-wide test suite, so we start the name of any Python step file with a prefix indicating what context of the system the step applies. Since feature files are allowed to be in nested directories, we create a directory for each context. Note that some steps can be reused contexts in which case a prefix indicating a shared context can be used.
Allure Report
Allure is a test reporting framework which can be added to the Behave command line run and configured in the Jenkins job—conveniently creating a dashboard like above. All you have to do is run something like this (npm run dev is set in our package file from above):
$ pip install allure-behave
$ npm run dev
Jenkins Job
$ cd behave-integration-tests
$ pwd
$ rm -R allure-results
$ rm -rf reports/*.xml
$ python3 -m venv env
$ source venv/bin/activate
$ pip install allure-behave
$ pip install -r requirements.txt
$ npm run dev
The last step of the process is to set up an automated job that can kick off the Behave test suite at an interval of your choice. For this, we recommend Jenkins because of the existing integration with Allure and Slack. We have configured our job so that it pulls our development branch, runs the test suite, and then Slack messages a dedicated channel with a link to the Allure dashboard daily. In order to run daily, we have a Build Trigger with the schedule “H 10 * * *”. The commands above are the actual set of Build commands for our Jenkins job. Finally, we configure Jenkin’s Allure Report Post Build action with a path to our generated Allure results folder and set a Slack Notification. After those are set up, then you are ready to reap the rewards of continuing insight into your system.
