Increasing Quality and Velocity with Continuous Testing

What is Continuous Testing?

As the pace of business and technology accelerates, the process of developing and releasing software needs to continuously improve. Quality needs to be integrated into the Software Development cycle as early as possible in the process. That’s where the Shift-Left model comes into play, which forms the basis for Continuous Testing.

The responsibility for testing and releasing needs to shift left in the software development lifecycle. Instead of passing code across multiple departments (System Design -> Development -> Quality Assurance -> Infrastructure and Deployment -> Security), Continuous Testing knocks down the walls and empowers engineers to do all of the above on their own, making quality everyone’s responsibility.

As you can see below, shifting left has a huge impact on the timeline and the quality of the projects you are working on:

Image from devopedia.org

The shift-left model emphasizes on quality from the beginning of the task, which increases the likelihood of finding and correcting issues earlier on in the process. For the shift-left model to be efficient, we need to focus on quality at speed.

Our Challenges to Achieving Quality

To ensure quality at speed, we at Ordergroove aim for a large number of unit tests, a smaller number of integration tests and an even smaller number of functional/UI tests. This follows what is called the Testing Pyramid.

From Test Pyramid: the key to good automated test strategy

However, with quality at speed also comes the challenge of on-demand testing, concurrent testing, test automation, test data/fixture management and test reliability.

This is how we solved these challenge and removed the QA bottleneck by shifting functional testing left in the development cycle.

More Context…

We recently launched a new product called Relationship Commerce Control Center (RC3) with the purpose of enabling retail brands to manage their settings and content for their Ordergroove integration.

RC3 is a front-end application written in React that integrates with our Python microservices. We needed to have a framework in place so that our engineers could easily and reliably add more functional tests.

Picking a tool

After doing extensive research, we came across a tool called Cypress which claims that “The web has evolved. Finally, testing has too. Cypress is Fast, easy and reliable [and] takes the pain out of front-end testing”, so we wanted to give it a try!
Here are the Pros and Cons we encountered while researching and testing the tool:

Pros:

• Runs in your browser: Unlike many other tools such as Selenium, Protractor or WebdriverIO, Cypress doesn’t need any special browsers or drivers to run. Cypress just runs your tests inside the browser and you can debug by inspecting elements as you normally would.
• Support for parallel testing: Cypress can run multiple test cases at the same time and speed up the time to run the full suite.
• Time Travel: The Time Travel feature allows you to go back to a previous step in one of the tests that already ran so that you can go back in time and inspect elements to troubleshoot something.
• Well-known syntax: Cypress uses a fork of Mocha as its test runner, Chai for assertions and Sinon for mocking, so if you’ve been writing tests in Javascript in the past, it will look extremely similar to what you’ve seen before. See video
• Automatic waiting: Cypress automatically waits for DOM or resources to load. We don’t have a single wait statement in our tests. No sleep statements or waitForPageToLoad nightmares!
• Assertion retry: Transparently retries assertions which eliminates a lot of the flakiness.
• Comes with a shiny local app: Local development comes with a GUI to run your tests and troubleshoot easily, but also has a CLI for your CI tool. Your CI tool can also record the run in a GUI for troubleshooting purposes.
• Screenshots: Cypress automatically takes screenshot upon failure when running using the Cypress CLI tool — An obvious one, but you’d be surprised how often this feature isn’t available out of the box
• Docker Support: Built-in support for Docker with public images that contain browsers, which play nicely with our Jenkins-Kubernetes CI/CD setup, and makes it very easy to run tests in a headless browser.
• Open Source Tool: Tool under heavy development that comes with an active community and very responsive developers

Cypress GUI (from docs.cypress.io)

Cons:

• Quite new: Cypress is relatively new (v1.0.0 came out in Oct 2017), so the community isn’t as vast as the Selenium community.
• Chrome or nothing: The founder and CEO of Cypress does not believe in cross-browser testing. The Firefox Support PR has been opened since Feb 2018 and has been de-prioritized multiple times. Not a problem in our case since we only want to fully support Chrome, but this will block adoption for many projects.
  Currently supported browsers are Chrome, Chromium, Chrome Canary and Electron
• No multi-language support: You can only write tests in Javascript or Typescript.
• No support for multiple tabs: Because it runs inside your browser, there is no support for tests where you need to access multiple tabs.
• Issues with shadow DOM: Need workaround to access shadow DOM
• No cloud testing support: There is currently no integration with BrowserStack or Sauce Labs… for now…

Demo:

See the demo video on the Cypress website: https://www.cypress.io

Building the framework

We saw more pros than cons for Ordergroove, so we went ahead and started using and testing Cypress.
We took these steps in order to get Cypress integrated into our deployment workflow:

1. Build a framework that abstracts out methods that will be re-used across all test cases such as login, saveCookies, accessIframe, etc…
2. Create a few simple tests that log you in, click multiple links in the navigation panel and test that all pages are loading successfully.
3. Using Jenkins-Kubernetes, build Jenkins Pipeline that spins up a virtualized cluster containing all the dependencies for RC3 to run, and bootstrap test data dynamically using a seeding script.
4. Integrate tests to CI pipeline to determine how flakey/reliable the tests are.
5. Identify repetitive steps that can be automated in order to speed up the tests (i.e. you might want to cache the base migrations instead or running migrations from scratch every time).
6. Create scripts to reset the test data locally to the same data set as the Jenkins Pipeline
7. Add more tests gradually!

Did this solve our problems?

First of all, the biggest challenge was to handle on-demand testing so that engineers could test their code independently of each other. Since the pipeline was fully virtualized with Docker/Kubernetes, the pipeline could be spun up on-demand along with all its dependencies so that it doesn’t rely on any existing environment.
This also solved the issue with test environment management, concurrent testing and test automation.

Second, when the virtualized pipeline is spun up, all databases are bootstrapped with just an empty database. From there, we run migrations and manage the data through code via a seeding script. That gives us the ability to update data via code, which makes test data management much easier.

Cypress has a lot of built-in features such as “Automatic waiting” or “Assertion retries” that eliminates flakiness in these tests, which in turn, helped with the issue of test reliability.

There are also many other helpful features that will allow us to parallelize and scale the tests moving forward, while still giving us all the tools we need in order to troubleshoot the functional tests.

Finally, it is also important to note that the functional tests are all written in javascript, and the syntax is extremely similar to the way we write unit tests. So the learning curve here is rather gentle, making it easy to shift functional testing to the left in the SDLC.

Conclusion

Continuous Testing is crucial to the long-term success of an application, and shifting QA to the left allows us to be one step closer to quality at speed.

In future articles, we will cover how to apply the same shift-left model to Deployments & Infrastructure or Security, which will improve our quality and velocity even more!