Is running automation testing scripts in production a good practice?

Your immediate answer will be NO because you may have worried about possible data corruption in production. Let’s understand the importance of Production testing a little bit and why it is important to do testing on production.

Testing in production is not only important but also critical as it allows testers to detect bugs in the real-world scenarios and to ensure if the application works the way it is expected to after the deployment. As per standard practice, usually smoke and sanity tests with happy path, that are carried out in the production environment. In an Agile environment mostly the developer makes the required changes in the application and deploys it on production without verifying it from the QA team just to keep application live and running, however, these frequent changes creates a lot of bugs in the application. In such cases, the application crashes hampering the overall user experience.

Some testing guidelines that must be followed while testing in a production environment.

  • Create your own real-world test data.
  • The naming convention of test data should be realistic.
  • Do not play with other existing user’s data.
  • Create your credentials to access the application.
  • Never try a load test on a production environment.
  • Test only if there is less load on the application.

Why we should do Testing on Production

The goal of Testing in Production environment is to ensure that the application is stable and runs the way it is expected to in the production environment. Daily production testing gives confidence that the application runs smoothly and hassle free.

  • Monitor the application’s performance in real-time scenarios where the test cases are not predefined, and the user keeps changing data.
  • Run edge cases in real-time to detect network failure, weak connections and interruption by call.
  • Monitor the API responses at a peak traffic.
  • Detect bugs and malicious attacks which typically go unnoticed while testing on QA environments.
  • Helps in maintaining the quality of the application for superior user experience.
  • Prevent disasters with better resilience and recovery testing. The application can recover from expected (chaos) or unexpected events without loss of functionality and data.
  • Testing with production data (it’s hard to replicate production traffic and data, making it difficult to detect every possible scenario).
  • It will eliminate the risk of frequent deployments on the production environment when performed on a daily basis, while you monitor application performance in real time.

Risks should be considered

Testing production systems is going to give the most realistic view of how things are looking and what can be exploited, however, several risk to testing production systems should be considered:

  • Reduction in system performance.
  • Keeping the application live and running.
  • No backup plan for your application runs the risk of loss of data.
  • No rollback plan when a release goes sideways.
  • Potential for denial of service.
  • Potential for email flooding via web forms with no CAPTCHA protection.
  • Risk of databases getting filled with junk data that can’t be easily removed.
  • Potential exposure of sensitive information.

Three major risk we focus while performing on production environment

Safety If something goes wrong and all data from staging is accidentally removed, you don’t care. If it happens in production, it’s… let’s say annoying.

Performance An extensive suite of tests can easily use >90% CPU on several servers for several minutes. You can’t afford slowing down production servers for a few minutes ten times per hour.

Goal The goal of testing is to prevent issues in production. If you find an issue when the application is already deployed, it’s too late. Moreover, reverting to the previous revision can be complicated (i.e. what if the new version created a column in the database, and meanwhile, you have new records where this new column was used?).

Written by

Senior Practice Manager # 16+ Years of Experience # Focused on QE Strategy, Digital Transformation # Focused on QE Innovation & Initiatives

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store