Testing in Production: It’s Time
In the world of deployment to the cloud, distributed applications and microservices, agile and DevOps methodologies, there is no longer a life cycle phase devoted to traditional functional testing. Much of the testing becomes automated, from unit tests to smoke tests to system tests, but we have less and less functional and exploratory testing, and no phase of the lifecycle devoted strictly to testing.
Moving fast is a mantra in modern software methodologies; we deploy without full hands-on testing but have the ability to rapidly revert to an earlier “good” version. In the process, we are leaving some aspects of the quality behind.
When that application is in production, we have to know when it is in trouble, and just what that trouble is, to understand the nature of the problem and have some data for diagnosis and repair. Even today, many cloud teams simply check whether or not their application is still operating using a continuous “ping” command, without information on performance or the availability of specific operations. In fact, all a ping tells IT and the application team is whether or not the server/instance is still running, not how healthy it is.
Ways to Test in Production
Teams need more information than that if their goal is to have a high-quality application. A collection of techniques known collectively as application monitoring is the way to do so. Application monitoring encompasses a wide range of testing and data collection activities. Synthetic testing lets teams record and execute tests representing how users might use the application. These tests can run at different times of day, and typically from different locations around the world.
Monitoring in production also incorporates real user monitoring or RUM. RUM is just what the term implies — in real time, it looks at user requests coming in, and what is returned in response. When you profile across a large sample of users, you get a good picture of how actual users perceive and how they use your application.
Last, monitoring can include tracing individual transactions through the different components and tiers of the application. By tracing a transaction, you can look at code pathways and see what code is being exercised during a user session. This is useful in finding defects and slow code, as well as understanding what features are more commonly used.
While major cloud providers such as, Amazon with CloudWatch and Google with Stackdriver, have their own tools that perform at least some application monitoring functions, independent software companies such as AppDynamics, New Relic, Dynatrace, and some TestCraft integrations provide more complete monitoring solutions.
Production Monitoring is Testing
Why is this a function of the software tester? It’s not only something a tester should be doing, but it also will be one of the two or three things that will make up almost every testing job in the future. Unless we want the professional practice of testing to gradually fade away, it must adapt to new ways of building applications and managing the application lifecycle.
Think of production monitoring as shifting testing to the right. We talk a lot about shifting left, where more testing occurs earlier in the development process. That’s fine and important, but testing is almost always incomplete when an application is deployed in a DevOps or agile team because new features are rushed to the production build. How do we make up the testing, especially if we are deploying regularly or even continuously?
Simply, we do so when the application is in production. We find ways to look for defects and poor performance after the application is serving real users. Using application monitoring techniques, testers can use real data to find and diagnose application errors quickly, and with information that is derived from the actual production environment.
Testing in Production as the future
The idea of testing continuing into production bothers many testers because testing has traditionally verified the quality of the application prior to release to production. However, thanks to agile and DevOps, testing is changing radically, and testers have to look at the best way to change. Engaging the application at all times during development and deployment is the most promising way of doing that.
Testing in production is in the future of testing. Testers will continue to do traditional test activities, especially exploratory testing. Automation has proven itself valuable in functional testing, by enabling testers to spend less time on rote manual testing. And AI will provide yet-undefined benefits across the testing process.
