How We Built Scalable End-to-End Tests

Morris Chen
Jul 2, 2018 · 9 min read

Two years ago, Bluecore started building UI products. Up to that point we had been good about creating backend unit tests, but creating full scale UI tests is a completely different organizational experience. End-to-end (E2E) UI testing is hard, and we ran into several challenges. Our tests were brittle, and debugging failures could be a nightmare. Tests were taking forever to finish or were too expensive to run often. When we searched for help online, we got mostly results with someone trying to sell us their silver bullet solution they claimed would magically make all our testing problems go away (but probably wouldn’t). In this article, I’ll share our experience with three different test frameworks and how well they helped us address these issues.

What do we mean by scalable?

It’s relatively easy to stand up some tests and build initial coverage for features. What’s significantly harder to do is build tests that stay reliable over time. There are several best practices we had to learn the hard way that led to successful testing. Examples include building idempotent tests, the test pyramid model, keeping tests small, and quarantining noisy tests. Sure, maybe whatever test infrastructure we created was great at first, but as we grew our test suite, problems started creeping in.

However well-intentioned we were to start, sometimes problems would begin to snowball and grow. Test results start becoming unreliable, which makes people lose faith in the tests. This leads to people spending less time working on the tests, which leads to unreliable test results, and then we’re stuck in a vicious cycle. Eventually the pain reaches a tipping point, and our team decides that it is easier to start over from scratch with a new testing approach than it is to convince everyone to fix the existing problems.

Image for post
Image for post
Not a recycling diagram

In order to spare yourself from the aforementioned testing death spiral, we recommend you consider the following: As you grow your test suite over time, are the following criteria fulfilled?

  • Easy to write: This leads to increased adoption and test coverage.

Here’s a high-level comparison of how we rated each of our E2E test frameworks:

Image for post
Image for post
*VS is used to anonymously refer to our Vendor Solution of crowd-sourced tests.

Now let’s take a closer look at each of these.

How we approached building UI tests quickly: Enter our vendor solution

The first versions of our UI were unreliable and buggy. A big reason for this is that we were manually testing our UI changes. As an early stage startup, we were hyper focused on finding product-market fit. This meant we had to build UI workflows quickly, and our test debt ballooned. With limited resources and no QA team, there weren’t many great options for building automated test coverage. However, we found a software testing vendor that could help us quickly create “automated” tests. I’ll refer to it anonymously as VS. The way VS works is that we write test steps in plain english and it utilizes testers from around the world to execute these tests. Yes, someone is manually running through our tests, but from our point of view the tests are “automated” because executing tests is simply an API call from our CI/CD service.

These tests are easy to write but have a higher marginal cost than traditional automated tests because rather than computing power, we pay humans every time we run a test. We needed E2E coverage ASAP, and VS made it easy to build this in a pinch. Our approach was to create new tests in VS before we eventually automated them ourselves.

Let’s see how VS does with our scalability criteria:

  • Easy to write: Yes — Anyone can write tests because it requires no technical expertise. The interface is intuitive and easy to use.

How we addressed the cost efficiency concern: Enter Selenium WebDriver

We never intended for VS to be our long-term solution for automated testing. Our strategy was that VS would be a stopgap to get “automated” sanity tests quickly. Meanwhile, we built a Selenium WebDriver test framework where we planned on migrating all of our VS tests. To summarize: any new tests for upcoming features could be created quickly in VS but would eventually get migrated to Selenium once the feature entered General Availability.

Let’s see how Selenium does with our scalability criteria:

  • Easy to write: No — Unlike VS, writing a Selenium test requires technical expertise. Only people with coding experience are qualified to write them. Furthermore, while we can be up and running with VS in just an hour or two, we found that our developers with no prior Selenium experience should be allocated 3–5 days to become comfortable with the tool and the page object model we set up before they could start writing their own tests.
Image for post
Image for post
  • Easy to maintain: Somewhat — We use Allure to notify Slack with test results. It shows us exactly what test step failed with screenshots and logs. However, while our test results are more informative from a technical perspective than VS, there still isn’t great support for identifying where in the source code the problems are.
Image for post
Image for post
Cost models of VS and Selenium: Selenium requires a bigger initial time investment but is cheaper over time because test execution is cheaper.

Our current focus: Enter Cypress

Although Selenium solves our cost scaling issue, there are two major pain points that are preventing higher adoption rates and test coverage from our development teams. Writing tests in Selenium is much harder than in VS, and it can be hard to troubleshoot false positives. During an engineering collaboration with another popular startup, our teammate asked how they addressed UI test case flakiness. One of their developers chimed in that using the open-source project Cypress was working well for them. This was the catalyst for us to investigate how well Cypress could solve our testing needs.

The main architectural difference between Selenium WebDriver and Cypress is that Selenium runs test scripts remotely to the browser, while Cypress runs within the browser. For a more detailed explanation of the key differences, check out this summary. After an evaluation period of two months, we concluded Cypress was the framework that best addresses our E2E UI testing needs thus far because it makes testing easier, faster, and cheaper.

Let’s see how Cypress does with our scalability criteria:

  • Easy to write: Yes — Like Selenium, Cypress requires technical expertise in order to write tests. However, installation is ridiculously easy so we can be up and running in a matter of minutes. Tests are intuitive to create, and Cypress provides great online documentation. The open-source community is also very active. When our developers have reached out for support, someone usually responds with an answer in a matter of hours.
Image for post
Image for post
Test tool cost comparison: I’ve hidden the actual dollar costs and used 1 bluecoin (not our cryptocurrency) to represent our Selenium cost as a baseline.

Learnings

There are a lot of testing best practices Bluecore is getting better at as we grow. Test results triage, fixing or disabling noisy tests, and writing shorter tests are examples of processes we try to improve regularly. Without this attention, we know that our team could lose confidence in our test results and our testing framework. On top of this, we came away with some other key takeaways:

  • While VS gave us a quick way to pay our test debt, it was a high interest loan. It’s hard to convince people to prioritize migrating passing tests to another framework versus new feature work.

Bluecore Engineering

Go behind the scenes of the retail marketing platform.

Morris Chen

Written by

Engineer @ Bluecore

Bluecore Engineering

Go behind the scenes of the retail marketing platform. Interested in working with us? Check out our careers page here: https://www.bluecore.com/careers

Morris Chen

Written by

Engineer @ Bluecore

Bluecore Engineering

Go behind the scenes of the retail marketing platform. Interested in working with us? Check out our careers page here: https://www.bluecore.com/careers

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store