Continuous Testing Drives the Failure of Engineered Test Creation

Venky Ganti
Mar 9 · 3 min read

Consumerization of the enterprise is a trend that has started about a decade ago. The rapid arrival of user-friendly consumer software has fundamentally changed user expectations from enterprise software in two ways. First, enterprise products are now expected to deliver user experiences aligned with the needs of modern enterprise personnel. Second, enterprise products are now expected to release at a much more rapid clip than before. However, the quality expected from enterprise software continues to be as high as ever. In this article, we focus on the challenges of achieving higher agility while continuing to retain enterprise-level quality.

Consumer software products such as Google search and Facebook Newsfeed deliver rapid changes very frequently. They can achieve this speed because they can fundamentally rely on their end-users for testing. The following three reasons enable them to release new versions to a tiny fraction of unsuspecting users and achieve comprehensive test coverage.

  • Their services receive a very large amount of traffic. Hence, they can “canary” their new releases — the process of releasing to a very tiny fraction of their users and get comprehensive test coverage and other top-level metrics.
  • Their end-users rely on these services for free and are significantly more tolerant of infrequent bugs. They can choose such tiny fractions of users at random from their large user base and hence no single user is going to be frequently subject to new & potentially buggy software.
  • Most of their services are read-only and hence it is easy to roll back changes. Bugs in the largely untested software don’t impact backend state and hence roll back only involves software and does not require “undoing” any state changes.

However, teams developing enterprise applications cannot even dream of such luxury. Enterprise software usually doesn’t have such high traffic, and users are not forgiving with respect to regressions in software quality.

Consequently, enterprise engineering teams are adopting Continuous Testing, often in the context of agile and DevOps trends. Therefore, these teams must test early and often, to detect regressions as early as possible in the development cycle. Such frequent and repeated testing necessitates the automation of all testing.

Due to pressures from release deadlines and the amount of effort necessary to create test automation, teams tend to only automate the happy and common paths and hope to return and backfill coverage gaps. In reality, that hope remains just hope and they do not have the time to proactively return and improve test coverage of any feature that has already been released.

Ultimately, these teams accumulate significant gaps in test coverage and hence the risk of regressions across versions. After some time when the complexity of software is sufficiently high, every release also comes with regressions. The cost of fixing regressions (root cause analysis, bug fixes, and patch releases) and the massive amount of effort for creating common path automation for new functionality together negate much of the velocity gains that were expected from adopting agile and DevOps practices in the first place.

If a large majority of enterprise software developments are to successfully adopt the Continuous Testing processes with enough coverage to deliver robust software, we need a fundamentally new approach which significantly reduces the amount of engineering effort to create tests.

We need an approach that

  • Requires low engineering effort, much less than that required by current engineered approaches
  • Achieves high test coverage to prevent almost all unanticipated regressions
  • Requires low infrastructure resources, especially not have the production scale databases to be live during testing
  • Can be used by developers in their commit workflow — run tests as soon as a PR is ready to be tested

At Mesh Dynamics, we are developing a solution that achieves the above goals for testing a broad segment of enterprise software applications. We are happy to learn more about needs and scenarios and are eager to help if possible.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store