UI Test Automation as part of Continuous Integration Process

Some time ago I was part of a debate about if it was helpful to have UI automation as part of the CI process to help determine if the build was healthy enough to start the test and verification process and eventually flag it as a release candidate.

A simple way to explain what is happening with our CI process

There were a lot of different opinions mostly leaning to ‘yes, it is helpful’. The only guy that was leaning to a completely ‘no, that is a terrible idea’ was one of the lead developers of the project. I agree that he had a good point: the environments we are using in our CI process are terrible unstable and slow.

After some discussion we all agreed that, for different reasons, we were going to invest time and effort to create our set of tests to run after a new version of the code is deployed to our CI environment. We call this our CI smoke tests.

The reasons were:

• Once the set of test cases are automated, they will work over and over again. Same tests against different version of the code.
• We can select the test cases based on the important features we need to preserve, no matter what we are changing in our code. For us, is really important that the user is able to add a product to the bag and start the check out process.
• These tests will rarely change, unless we change something in the design. With good communication across the teams we can work in the design change soon.
• Will help us train new developers getting on board. We have seen that one a new member joins, there will be a broken build.

We acknowledged that there were work to do before having a set of smoke tests running with every deployment. We realized that these steps we needed to take to get to a healthy UI Tests in the CI process:

• Participation from all teams to select the test cases to be part of the smoke test suit is required. When we had our meeting with Business team gave a good input on giving information on what was more likely to break.
• The tests must be short and meaningful. A developer does not want to wait for 30 minutes to see if a build is broken or not. We agreed that 5 minutes were good enough.
• We needed to work on speed up our CI environment and support concurrent users. System Administrators and DevOps team were really helpful in this point.
• We worked on our process on how to handle a broken build. No more commits from developers, QA assessing the cause of the failure and communicating to all the team.

It was a big change in the way we were doing our CI and took us a while to get it going.

Since we implemented this process about a year ago, we have stopped worrying about making sure if the user was able to add something to the bag, if the user was able to search, if the user was able to navigate to the most visited pages and if pricing was showing correctly before we received a new build to test.

Now we worry more about how to keep up with the large amount of new features changing in a short time.