Evolution of web testing strategies — stop the monkey business

In this blog post series I want to go through our journey of creating QA-less Testing Strategy for the Nielsen Marketing Cloud frontend application that we’ve been building in the last couple of years.

In Nielsen Marketing Cloud R&D we are operating without any manual QA to test new features and regressions for each release and it provides a lot of benefits:

• manual QA is expensive. For automatic tests you write a test once and run it a million times;
• each developer feels more responsible over what he/she is doing, because there is no one who will check what he/she developed;
• reducing time to market, you can push new features each day without relying on additional teams/people;
• ability and confidence to refactor your code;
• self documentation for your code.

In the long run, good test automation gives you better quality and reduces the amount of regression defects for old functionality.

But that is the million dollar question — how to do effective automated tests?

Act I — Bad Practices

When most developers hear “test automation” they think “automated user behaviour” (a.k.a End To End selenium tests) and we were no exception to this rule.

Of course we knew about the importance of unit and integration tests, but “Let’s go into a live system, click some buttons and it will check all of the layers together” was a pretty logical first step to go with automation.

So a decision was made: let’s write a lot of end to end (e2e) UI selenium tests!

It seemed like a good idea, but the result we achieved had the following issues:

1. Unstable tests

At some point, when we started to accumulate big amounts of e2e UI tests, they started to fail. And the failures weren’t deterministic.

What did our developers do when they didn’t understand why the tests randomly fail?
Correct, they added sleeps!

It starts with adding sleep(300) in someplace, then sleep(1000), then sleep(3000).

All of a sudden the test code is a huge slumber party! with ice cream and everything! (pun intended, explained later).

And at some point we’ve found in our tests something like this:

sleep(60000) //wait for action to be completed

I hope it’s pretty clear why sleeps are not the best solution, right? :) *

* sleeps are slowing down your tests and they are constant, which means that if something will happen longer than you expected — your test will fail, so tests will be not deterministic and not stable.

2. Tests are running very slowly

Because the automated e2e suite tests the actual system with a real backend and database, running them becomes a really slow process.

When something is slow — developers will usually avoid it, so they will not run them locally and push code with the hope that the CI will be green somehow. And then it’s not. And our CI is red. And the developer went home already. And …

3. Test failures are not informative

One of the biggest problems of our e2e UI tests is that when something is going wrong, you don’t really understand where is the problem.

And the problem can be EVERYWHERE: environment (remember, that ALL system layers are parts of equation), configuration, backend, frontend, data or just an unstable test (not enough sleeps).

And only one thing you see:

In the result — the investigation of what really failed, takes a lot of time and becomes annoying.

4. Data for tests

Because we are testing our system with all of the layers in place, it’s very hard to imitate or manufacture complex scenarios. e.g. if we want to check how our UI will react when the server is down or the data in DB is corrupted — it is not a trivial task, so most of developers just test the positive scenarios and no one knows how the system will behave when there will be a problem (and there WILL BE a problem).

But this is just an example, you often need a very specific case to test that a specific UI element is disabled and making all the stars align just for this specific scenario takes a lot of effort. Not to mention cleaning everything up and setting it up again for the next test.

5. Dynamic UI

In modern SPA web applications the UI/UX is very dynamic, elements appear and disappear, animations hide elements, popups, asynchronous changing parts of UI based on data in different parts and a lot of other dynamic stuff.

This makes automation of flow much more complicated, because in a lot of cases you don’t really know what is the real condition for some action finishing successfully.

6. Let’s build our own framework!

At some point we saw, that our tests are pretty repetitive, because different parts of our UI have the same elements (searches, filters, tables).

We didn’t want to copy paste the same test code, so we started to build our own testing framework with some generic Test Steps, which can be reused in other Test Cases.

And then we figured out, that the same UI element can be used differently (single/multi select, flat/hierarchy table, etc.), has different data and so on. So our generic test cases were growing up with configuration, inheritance, factories and other coding patterns.

In the end our code base for tests became really complex and only few people in our team really understood how it worked.

Of course, some of the problems can be partially solved (parallel tests running, making screenshots on failures, etc.), but the result is that the e2e UI testing solution becomes very problematic, hard to maintain and really expensive to support.

Moreover, at some point we had so many false alarms and stability problems, that developers stopped taking “red build” seriously, which is worse than to not have tests at all!

In the result we have:

• Thousands of e2e tests
• Takes 1.5h to run (~20 mins in parallel)
• Fails almost everyday, and when they are red — need to do manual sanity/regression
• Takes a lot of developer’s time to reproduce and fix tests
• 30k LOC of e2e tests

The end result of our attempt to add test automation manifested to the Ice Cream Cone AntiPattern in our happy happy party…

So we decided to drastically reduce the amount of e2e tests, and to use something much more appropriate instead. All told in the next act.