UI Automation Testing: is it Worth it?

The Question

I have been developing web applications for a while now, and the topic of UI automation testing has come up at every company I’ve worked for. Mainly, what I have seen is either avoidance or failed half-hearted attempts.

The usual reason for not doing UI automation testing I hear is essentially that “the juice is not worth the squeeze.” The common perception is that it takes a great deal of energy (read: time, money) not only to kickstart a UI automation practice, but also to keep the wheels greased.

Despite my first-hand experience, I have also heard bits of various UI automation success stories. Unfortunately, it’s hard to know if many of these stories are tainted by either effort justification bias or marketing goals.

And this leads me to the core question: how often is UI automation really worth the effort, and under what conditions? What are the keys to getting a positive ROI in those conditions?

Spoiler alert: this entry doesn’t contain any answers. This is the beginning of a multi-week investigation, and my goal in this installment is just to form the right questions and start gathering data. Can you help me?

Hypothesis

I believe that more often than not, UI automation testing really can yield a positive ROI for a long-lived application.

However, I think the problem in the industry is that there isn’t a widely accepted blueprint for doing UI automation testing right. Companies have certainly succeeded internally with automation solutions, but they have not gone the route of packaging their blueprint and custom software to offer it to the outside world. At least, they have not done so effectively.

This lack of a clear path, combined with an often misrepresented view of the true cost of manual testing is the reason we don’t see a level of successful automated UI testing adoption that we see for automated service testing. I think that the answers we find will point us toward a “when” in addition to “how.”

My goal for this investigation

Let me be forthcoming that I do have an angle besides my own curiosity. Based on your stories, I would like to determine if there really is such a widely applicable UI automation testing blueprint. If it exists, then I would like to share it back with the world.

Some light priming

The next section has a list of questions pertaining to UI automation success stories. Some of the questions are related to costs of software quality, so before I ask, I would like to get everyone on the same page about the true costs of software quality.

To paraphrase several concepts from this excellent presentation:

Cost of Software Quality = Cost of Control + Cost of Failure of Control Costs

Cost of Control represents the sum of all costs involved in quality control. This includes:

• Staffing resources.
• Managing and educating resources.

Cost of Failure of Control represents the sum of all costs associated with correcting for failures in control. This often includes:

• Short-term reductions of sales due to software errors.
• Long-term reductions of sales due to damaged reputations as a result of software quality.
• Increased development costs as a result of extensive late bug detection interruptions.

Do you have a QA automation story to share?

Please email me directly. I can keep it as confidential as you like, but I will default to not sharing any company names. Here is an initial list of questions I’m interested in:

1. Where does UI automation testing fit into your organization or project?
2. Did adoption of QA automation generate a positive ROI?
3. Who writes and maintains the tests? Is it developers or dedicated QA resources?
4. How are the test cases and results made visible?
5. Do you have any specific tips for adopting a successful UI automation practice?
6. Do you mind if I follow up with you to ask a few more questions?

I sincerely appreciate any time you can spare to share your story. My promise to you is that I will diligently analyze the stories I gather and follow up with another post.