Laying the foundation of stable web testing

Photo by ZMorph All-in-One 3D Printers on Unsplash

Testing is an integral part of our software development lifecycle and teams nowadays rely heavily on automated tests to deliver value faster. Although it sounds very shiny at the beginning, it comes with a cost.

So, how many times have you heard these statements?

“Developers don’t trust the automated tests”

“Our automated tests are flaky and breaking the CI builds”

“Let’s delete the tests or skip them”

If you have in any way been involved in the process of test automation, I bet at least once!

Here in Glovo, we were experiencing similar problems in our frontend applications’ testing and a few months back the newly formed Web Platform team decided to take a stab at it!

When we started

We were already using Cypress to run our UI automation tests and I must say it served the purpose very well. Therefore, we did not spend any time analysing any other tool. We also had a large number of automated tests for three of our front end applications already running on the CI.

But, here comes the challenges:

• Tests were not using the real-world representation of browsers because they were running on Cypress’s default browser — Electron
• There were frequent fire fighting with CI builds because backend data was unpredictable thus making the tests unstable
• These resulted in developer unhappiness as they spent significant time fixing the issues
• It yielded low scores in platform surveys because product teams were not happy with the test ecosystem
• Sometimes, there were gaps in test coverage because tests were being completely skipped from the CI
• There were more instances of anti-pattern than design-pattern because test code was not treated the same way as dev code
• There was no uniform solution for the issues because out of the three applications, one is a mix of Client-Side Rendering and Server Side Rendering

The Fixes

We figured out that the test failures are often due to the data changes but apart from that, debugging them was equally challenging due lack of good coding standards. To mitigate these issues we came up with two categories of fixes

Tests infrastructure stability

Here, we focused on providing a reliable platform for our test automation, by addressing some of the existing issues as follows:

1. Mocking Strategy

Currently, we do not have a separate environment with consistent data that can be leveraged for automated tests. Additionally, since our tests mainly focus on the functional aspect of the application, we wanted to keep them agnostic of the backend. Hence, we decided to mock all the requests. We achieved it in two ways for each of our architecturally different applications:

• Glovo Mock Server (Client + Server-side rendered applications) — We developed a bespoke mock server as we could not leverage Cypress mocking because of the hybrid nature of the application. The mock server is an express based server with intuitive REST APIs to update and reset the mocks. It stores a default state of mock responses for all endpoints. This enables us to test scenarios requiring various data for the same endpoint without relying on a dynamic environment. It has features to record mock responses on the fly when writing new tests but also enforces mocking for actual test execution by throwing errors for non mocked endpoints.
• Cypress Mocking (Client-side rendered applications) — For our client-side rendered applications, we use Cypress intercept to mock all common endpoints at the beginning of each test, thus creating a virtual mocked environment. With the new version of Cypress, we can override the existing mocks whenever a test needs a new response for a specific endpoint. Just like our mock server, we can record responses on the fly and enforce mocking by throwing an error for any non-mocked endpoint.

2. Using Github Actions for tests

We moved from Jenkins to GHA to run the tests as a part of PR and Main build checks. GHA has out of the box integration with Cypress and handles the parallel execution of all the tests on different viewports on Chrome. This has reduced the build time, increased the coverage and enhanced reliability.

3. Test Observability

We leveraged Datadog metrics to keep a watch on flaky and failed tests for our CI executions by developing a bespoke Cypress DataDog plugin and integrating it with the framework. As a process, we visit the dashboard and monitor the tests' flakiness regularly, which helps us identify issues before they become blockers.

Tests development experience

Here, our goal was to provide best practice guidelines and utilities to make test development easy for engineers. Thus, we worked on the following areas to achieve it:

1. Design Pattern

Our existing framework was already using the Page Object Model but we observed that was not followed diligently. Since it is one of the most successful design patterns in test automation, we stuck to it only with more refined guidelines and strict checks. Although, Cypress advocates for app actions, we believe it is not scalable enough for larger frameworks. So, to keep it simple, we used JavaScript modules for our Page Objects. Our motivation was to ensure that the tests in the spec files are clear and readable with no complex logic. We wanted to avoid situations where our test code itself needed tests to ensure they worked fine (Sounds like a vicious circle!). We defined our Page Objects based on how the user perceives a page or a section of the application and are not influenced by the application code. The Page Objects have functions that should return an element of that page or perform a set of user actions on the page, therefore all assertions are in the spec file for better readability of the tests.

Take, for example, a Store Page in an e-commerce website that has sections and subsections with text, images, buttons and a feature to search for items. The Page Object module for this page should look like this:

const title = () => cy.get('.someReusableTitle')
const catalogueButton = () => cy.get('.someReusableCatalogue')
const searchBar = () => cy.get('.someReusableSearchBar')
const itemInSearchResults = (item) => { 
    searchBar().click().type(item)
    // code to search for an item
    return true // if present based on the search result
    return false // if not present in search results
}
const StorePage = {
    title,
    catalogueButton,
    searchBar,
    itemInSearchResults
}
export default StorePage

From the coding perspective, the page could be built using many reusable components, but from the user perspective, it is a unique page that gives information about the store. Thus, every such unique page of the application should be defined as a Page Object module. One thing to keep in mind is, if a section is common across the application, it should be defined as a unique Page Object. For example a header, footer or top navigation.

A couple of things to notice in the above example are:

• the Page Objects return the elements of the page
• the Page Objects perform a set of actions on that page that either returns a boolean or a value, based on the result

However, the second point should be minimised. Most of such operations should be a part of the tests to make them more readable. If the set of operations is performed frequently enough in multiple tests, only then we can decide to create such a reusable function in the Page Object. This rule is primarily to discourage the overuse of the DRY principle.

Additionally, we are restricting assertions in the Page Objects. The assertions must be added as a part of the tests to make them clear and readable. Thus, the spec file should look like this:

import { StorePage } from './pages/store.page'
const storeName = 'Example Store'
const item = 'Some Item'
describe('Store page', () => {
  it('user should be able to search for an item', () => {
    cy.visit('/store-page')
    StorePage.title().should('have.text', 'Store Name')
    StorePage.catalogueButton().click()
    const itemAvailable = StorePage.itemInSearchResults(item)
    expect(itemAvailable).to.be.true
  })
})

If you read through the test, it should give a clear understanding of all the different steps that it is performing.

2. Custom ESLint rules

Apart from data consistency, we wanted to ensure code good quality for our tests. We decided to automate checks for all possible cases to ensure that the coding guidelines and design patterns were followed correctly. Therefore, we came up with the following custom ESLint rules:

• No new function declaration in spec files- this enforces reusability by not allowing any new functions declaration in the spec files. This ensures that functions are written as a part of the Page object or helpers.
• No element querying in spec files- this enforces Page Object Model by restricting any element querying in the spec file. This ensures that the page locators are defined within the Page Objects Modules.
• No assertions in page object functions- this restricts functions in the Page Objects to have assertions, as we believe that having the assertions in spec files result in more readable tests which are easier to debug.
• No cypress intercept allowed- this restricts the use of cypress mocking completely in our hybrid application as we wanted to support a uniform technique. This rule is not used in our client-side rendered application.
• The maximum describe level allowed is 1- this enhances the readability of the tests by keeping the maximum allowed description level to one.
• No cy.visit() in before() hook- this ensures that every test navigates to the URL either inside the test or in a beforeEach() block and thus they are atomic or independent and do not rely on the state of previous tests.

3. Migration of critical tests

As a platform team, we provided blueprints that follow all set guidelines by rewriting some of our applications' critical paths’ tests. These tests are meant to be the gold standard that can be referenced by product teams.

4. Code Ownership and Pairing for advocacy

We added ourselves as code owners (Github CODEOWNERS file) for all the test code and started doing a thorough PR review for changes related to it. This has helped in evangelising the best practices. Many such PR reviews resulted in a pairing session with the product engineers which helped us share the knowledge and advocate our strategy to the target audience.

All the improvement work done by the Web Platform team over the last couple of quarters has laid the foundation to create a robust and scalable testing ecosystem for our front end applications. The results have been promising so far with rare build outages and increased developer productivity. To give an example, the main build failure rate for our primary application has decreased by 72% and main build time has reduced by 24%. These improvements have encouraged the product teams to enable automated tests in their CI pipeline and add more test coverage with confidence. It has also paved the path for the platform team to introduce stronger safety nets in the form of visual and accessibility testing.

It’s a long road but it’s worth it!