How Automated End-to-End Tests Give Us Confidence and Make Us Brave

Pushing code into an application that manages millions of tonnes of industrial materials can be a nerve-wrecking experience, especially if you are new to the company. In this post we will take a look at how we use automated end-to-end tests to increase confidence among our developers and help us deliver a stable product to our end-users.

This post is a part of a series of posts about our Enabling Team; if you have not already, you can read the introductory post, The Emergence of the Enabling Team.

Historical Perspective

Quality Assurance has been a central part of product development for centuries now, and traditional software is not all that different from physical products in that sense. Online software is a little different as unlike the old CD-rom, it changes even after it is in the hands of the customer.

In its essence, a test is a series of actions that are measured on whether or not they have the intended consequences in the application. Earlier these steps would be written down in lists and performed manually, but now we can automate these in scripts that act as a user of our application.

There are many benefits to this, and a few challenges. Automated tests are unambiguous and executive. They serve as documentation and unlike traditional documentation, they cannot get out of sync with the application. Furthermore, we can run them whenever we want and as often as we need. The challenge is that you will have to instruct a program how to test instead of instructing a human being. This means writing code.

Testing in Autonomous Development Teams

As each of our development teams are responsible for their features from cradle to production, implementing these tests also becomes the responsibility of the team. The developers that build it have the best starting point for understanding the intended behaviour of a feature, and as the only certain thing in web development is change, they will also know how to change the tests when improvements are made to existing features.

The Enabling Team

Testing and release is a part of our Enabling Team (you can read more about it here), a team of experts in key areas of software development: testing, DevOps and security. The role of the testing enabler is to make some decisions on behalf of the teams in order to reduce the complexity and integrate the testing system into our existing pipelines. Below are three important areas that the enabling team are working on:

Running all the parts

So far it sounds pretty straight forward, you get some program to run through the application and check that actions results in the predicted behaviour. This is where it gets complicated though. Many web applications, including ours, consists of multiple, independent components.

In order to verify that all components are integrating correctly with each other and that we’re delivering the product as promised to our customers, we must run all of these parts together in a configuration that mimics our production environment, in order to gain the confidence that we want.

In order to perform functional testing, we run all the components in a way that mimics the production environment, but isn’t actually the production environment.

There are some things we can allow ourselves to leave out and not mimic / replicate, such as some cloud settings, EG. firewalls and virtual networks. An erroneous configuration in any of these would likely cause the whole application to become unavailable, which would be picked up by other availability tests.

Hence, running the application and all of its components can be done easier when done for the purpose of functional testing. Specifically we utilize docker, docker-in-docker and docker-compose for this.

Integrating Into CI

This reduced environment is assembled on ephemeral VMs in parallel, every time somebody pushes code to any of the repositories. After successful functional tests, the application is deployed to a test environment, which mimics production very closely. Availability tests are run against the test environment and if everything goes right, then the application is further deployed to production.

Local development experience

This above-mentioned reduced environment is much more light weight than our Kubernetes environment and is more suited for running locally (synergy). With this, every developer have the ability to make and verify changes to a whole application vertical before committing anything.

We’ve also developed utilities to allow developers to generate and populate their local environment with semi-realistic but random data, allowing them to develop the application without resorting to using sensitive production data to visualise changes.

Advocate Program

As each team has to take responsibility for the test coverage of their features, one of the most important tasks for the Enabling Team is to offer training and guidance to the developers in the teams. Our Advocates Program aims to provide selected individuals from each team with some additional insights into one of the areas of the Enabling Team. In the case of testing, this could be getting the mindset of testing, looking closer at the best practices for implementing tests or learning more about how the test-rig is configured.

In addition to all of the above, the Enabling Team helps our development teams structure the process of defining the intended behaviour. This is also a process that can be more complicated than you imagine going in, and the devil is always in the details.

What It Means For Us

Psychological Safety

The most important thing we get from having a solid automated end-to-end testing system implemented is that we can reduce the anxiety levels connected with the production of new features, and the updating of existing features.

Removing a bit of this uncertainty helps us increase the speed of delivery, and allows us to confidently make changes where we would otherwise have been hesitant to touch something.

The Bottom Line

All we know for sure is that we need these tests, and that they have an invaluable contribution to our ability to develop as fast as we do. Every day we are learning new things about how to structure the process, and how to solve paradoxes between best practices in different fields. We are all just trying to figure out how to do these things, and this is what makes it so interesting to work with.

About The Author

Jonas Amundsen is a member of our Enabling Team (read more about it here) and with his background as a full-stack developer he supports the technology department in a range of areas including testing and release where he has had a leading role in the development of our current setup. After getting his degree in Computer Science from the Norwegian University of Science and Technology he started his career with Itera and Netlight before founding his own company and joining Klaveness Digital as a consultant two and a half years ago.

We Are Hiring

Do you have a passion for DevOps, Security or Testing? We are currently scaling our Enabling Team and would love to hear from you. Visit our career page to see our open positions and to learn more about life at Klaveness Digital.