How to overcome test flakiness with web UI tests?

If you have been active in the testing space, one of the common concerns folks have around testing front-end applications is flakiness caused due to network latency or changes in backend APIs which prevents you from testing the front-end changes. A simple answer to this problem could be to mock back-end responses. But then, what if there are changes made to the backend APIs? That would still require human intervention to update the mocks. What if I told you that you could automatically generate these mocks to ensure that your tests consider the latest back-end changes?

At Hippo, when Susmitha Sethu from my team setup the Cypress framework using mocks for adoption, one of the questions that came up from the team was — “How are we going to handle the continuous changes made to the backend?” Susmitha then worked on a proof of concept that involved leveraging the @graphql-tools/mock package to auto-generate these mocks which further made our test framework more extensible and robust.

Let’s do a quick walkthrough of what libraries we used and how we implemented the auto-generate mocks capability into our test framework.

Pre-requisites:

Our backend framework uses GraphQL. You can also use GraphQL on top of your existing infrastructure: REST, SOAP, existing databases, or any other tech stack.

Web Test Framework Architecture

Here’s an overview of our test framework architecture using Cypress and the graphql mock package:

As Hippo is growing into a mature public company, we want to move faster but at the same time want to ensure we maintain a high-quality bar for the products we deliver. Speed and quality can be achieved only when we have a reliable automation framework which involves mocks.

Once a pull request is created, it triggers the sanity test suite which in turn uploads the Cypress test suite image to BrowserStack which is our 3rd party test lab to execute our tests. When the tests fail, pull requests are prevented from merging. This in turn ensures that the author of the pull request debugs, fixes the test failures and re-runs them. When the tests pass, the pull request is marked as ready to merge. We have implemented this process for a couple of projects and plan to expand it to other projects as well.

Why do we resort to mock responses for test automation?

Mocks are used in front-end applications for two key reasons:

• Develop features where backend is still in development
• To isolate and test UI changes

For both scenarios the mocks use the relevant data that is necessary for testing the component.

But for automation, we need a mock with all the required attributes to write E2E tests. As the product scales, managing and maintaining mocks manually becomes cumbersome.

Tools like @graphql-tools/mock package help auto-generate mocks and reduce the dependency on managing mocks manually.

Why use the auto-generate mock approach?

There are cases where the front-end doesn’t manage its own types and relies on auto generated types from back-end. In such cases, for any change in the back-end schema, failure to update mock schema will result in test failures as the front-end app is now interacting with stale mock response. This can be mitigated using auto-generate mocks which always returns data based on the updated back-end schema.

Example without auto-generated mock:

export const mockPolicy: Policy = {
  id: ‘abcdstuuid’,
  policyNumber: ‘ABC-9999999’,
  effectiveDate: ‘2020–09–23T00:00:00.000Z’,
  bindDate: ‘2020–09–23T00:00:00.000Z’,
  status: PolicyStatus.Canceled,
  expirationDate: ‘2021–09–23T00:00:00.000Z’,
  documents: [],
  currentPaymentMethod: PaymentMethodType.CreditCard,
  paymentStatus: PaymentStatus.Pending,
  presentationPaymentStatus: PresentationPaymentStatus.AchPaid,
  termsAndConditions: {
    areTermsAccepted: true,
    acceptTermsLink: undefined,
  },
};

Example with auto-generated mock:

mockGraphQL({
  CreditCard: () => ({ expiryYear: 2030, expiryMonth: 6 }),
  Payment: () => ({ dueDate: ‘2021–06–09T00:00:00.000Z’, amount: 420}),
  Charge: () => ({ chargedAtISODateString: ‘2021–06–09T00:00:00.000Z’}),
  DateTime: () => ‘2021–06–09T00:00:00.000Z’,
  PolicyStatus: () => PolicyStatus.Pending,
  TermsAndConditions: () => ({ areTermsAccepted: true }),
  PaymentStatus: () => ‘Pending’,
  Policy: () => ({
    cancellation: null,
    currentPaymentMethod: ‘CreditCard’,
    expirationDate: ‘2023–06–09T00:00:00.000Z’,
    paymentFrequency: ‘Annually’,
    policyNumber: ‘2222222’,
  }),
  PresentationPaymentStatus: () =>   PresentationPaymentStatus.CreditCardPaid,
});

Customizing mocks

For specific needs, you can further customize the mocks to return user specified data by using mock object(mockData) as shown below:

const schema = makeExecutableSchema({ typeDefs: schemaString });
export const schemaWithMocks = (mockData: IMocks<any>) => {
  return addMocksToSchema({
    schema, mocks: mockData, preserveResolvers: false,
  });
};
export const postAnyQuery = (mockedSchema) =>
cy.intercept(‘POST’, customerPortalBFF, (req) => {
if (req.body.operationName === ‘IdentityVerificationHash’) {
  req.reply(identityVerificationResponse);
} else if (req.body.operationName === ‘RequestCustomerContact’) {
  req.reply(requestCustomerContactResponse);
} else if (req.body.query) {
  graphql({
    schema: mockedSchema,
    source: req.body.query,
    variableValues: {
      id: ‘1234’,
    },
  }).then((result) => req.reply(result));
}
}).as(‘postAnyQuery’);
export const mockGraphQL = (mockData) => {
  postFlagship();
  postEvents();
  postStripe();
  postIdentity();
  postAnyQuery(schemaWithMocks(mockData));
};

Auto-generated Mock Example:

If we want to set a specific field e.g. claim contact in auto-generated mocks, then we just add it to mock object instead of mocking literally every attribute for every query.

mockGraphQL({
  DateTime: () => new Date(‘2021–07–01’).toISOString(),
  TermsAndConditions: () => ({ areTermsAccepted: true }),
  PresentationPaymentStatus: () =>   PresentationPaymentStatus.CreditCardPaid,
  ClaimContact: () => ({
    firstName: ‘xxx’,
    lastName: ‘xxx’,
    phoneNumber: ‘xxxxxxx’,
    email: ‘xxx@xxx.com’,
    __typename: ‘ClaimContact’,
 }),
   ClaimList:() => ({
    count: 0,
    items:[ {
    id:’0Zk6s000000CbZ4CAK’, claimNumber:’XXX- 0009894',status:ClaimStatusEnum.NewFnol,peril:ClaimTypeEnum.ServiceLine,
 }],
}),

Overall, we have achieved better efficiency, wherein we spent 1–2 hours each day debugging and fixing failing tests due to changes in back-end schema. By leveraging the auto-generated mock approach, we have optimized for that time spent debugging and increased reliability and confidence in these front-end tests.

I hope these insights from the Hippo Test Automation team have piqued your interest on improving stability and reliability of front-end automation!