How to build a test automation framework that will be used by the whole team

Pretty often you can see test automation framework successfully running tests and reporting results but not doing what it’s supposed to do: providing a reliable way for team members to build automated tests, and get reliable results.

This often happens when test automation framework is built without planning in advance and understanding how it will be used.

At first, the team realizes that they need automated tests. One of the engineers decides to take care of it (or gets assigned) — using the tools they are familiar with, they automate the first bunch of tests.

Since initially it’s a proof of concept, some things are being implemented via the fastest and most obvious solution, which is not always utilizing industry’s best practices. Such solutions introduce technical debt. If not addressed early, the impact of technical debt grows once framework is expanded.

As a result, few iterations later, team gets a test automation framework that can pretty well run tests that were in mind of author building it. But making a step aside, expanding coverage to additional features, or trying to get other engineers owning tests creation via such framework becomes a challenging task.

As an SDET, who has been often brought to build test automation frameworks from scratch or improve an existing solution, I see my mission as next:

Create reusable and maintainable test automation tools and frameworks that can be used by any engineer to build reliable tests with great code coverage, aligned to business needs

So, let’s discuss what it takes to build a good framework for a team.

Planning

During the planning session, it’s important to cover several areas.

Alignment to business needs

The first and one of the most important steps of building test automation framework is to analyze business needs and ensure that tool will follow them. Sit with team or product owner and go through next questions:

• What do we want to achieve with this test automation framework?
• How will it help the team?
• What are the biggest pain points that can be addressed by adding a test automation framework?

Tools

There is more than one tool for almost every task. Choosing the right tool might be a game changer and has a tremendous influence on how easy it will be to maintain and expand automation with frameworks later.

• What’s the list of tools that can potentially address the problem?
• How popular is the selected tool? Is it an industry-wide standard solution?
• Is there a new trendy tool that can be used for a problem?
• Does tool provide an out-of-the-box solution for typical issues?
• Is tool customizable? Will you need customization to fulfill your needs?
• Is there support and/or community associated with the tool?
• Can it be easily integrated into the existing system?

Based on these questions you should be able to come up with tools that should a right choice for you.

Ideally, it would be a popular tool, widely-used in industry, with good support and lots of existing solutions, which can be easily integrated into an existing system.

It also should be a tool that team is familiar with — unless team agrees that benefits of using this tool are worth spending additional time to learn it.

Team

This is a part of tools selection but it’s so important that I’d like to describe it in a separate section. A good test automation framework can be easily used by any engineer to create and modify tests for the feature they are working on. Once again:

Any engineer on the team should be able to write tests for their feature, using a test automation framework

This means that while choosing a tool and planning automated test framework development it’s critical to analyze next areas:

• How will the team be using the framework?
• Will they be responsible to write tests from scratch?
• Will they be required to modify tests when pushing a change to software?
• Is this tool aligned with team’s expertise?
• What efforts will be required from an engineer leading framework development to enable and support framework usage by other teammates?

Scope

Keep in mind that not everything can be automated. Basically, you want to automate:

• Stable repeatable scenarios with predictable results
• Integration of software with external services
• UI (to a reasonable level)

There are areas that may have lower ROI if being automated, or should not be covered by automation tests at all:

• New features with not finalized requirements
• Areas that heavily depend on external services with not predictable results
• Flaky areas
• Any functionality that requires human interaction which can not be reliably emulated

Define the scope based on business needs and biggest pain points that team currently has. Make sure that there’s understanding of how improving these areas will help the team, and everyone agrees that this is the right approach.

Designing test automation framework

Design is a phase of development that defines how easy it will be to use and maintain the framework, what needs will it fulfill, and how it will be implemented. Spending additional day on design will save weeks later.

There are several questions that help during Design phase.

How will the framework be used?

An approach that is recommended during the design phase:

Rely on how the framework will be used, not only what it will be doing

As mentioned above, a good framework should be reusable and maintainable. It’s critical to discuss with the team how it will be used, and account for all needs while designing framework architecture.

What framework will be doing?

Good design allows addressing potential issues and caveats before the team faces them. Understanding of those issues comes with experience. So it’s critical that engineers with most expertise get involved in an early stage of design. It’s also beneficial to rely on experience community and other teams.

Do not reinvent the wheel, use best practices and existing solutions

Following this approach helps to understand potential issues in advance and take measures to address them.

What problems will be solved by the framework?

So many times I’ve seen a similar feature of test automation framework being independently developed by several teams. Because of broken communication, engineers were not aware of other teams efforts and had to go through investigation phase on their own, looking for solutions and analyzing potential issues. Often they would come up with similar or identical solution.

Sync to other teams in company to see what problems they had and how they solved them

There’s a long list of challenges that will be faced by any team in the company building a framework from scratch. Solutions for most of them will be applicable to any team’s framework.

• Authorization
• Logging
• Debugging
• Reporting issues
• Handling test data

Syncing to other teams early and analyzing existing solutions and challenged allows to save lots of time and helps get the right direction.

Implementation of test automation framework

Often teams decide to use a more lean approach to building test automation framework than in building a product. This results in a lower quality of code. decreased stability and poor performance. It also leads to a framework being a “grey area” — when the team knows that it does something but most engineers don’t know what exactly is being done and how valuable are the results.

I believe this is a bad approach.

Treat the framework as a part of your product

Treat framework exactly as you would treat a new feature for the product. Each feature should go through analysis, design, planning and implementation phase. Same requirements for code quality and architecture should be applied to a framework.

Do code review

Code review is an amazing way to improve code quality and help engineers grow. While developing a framework, do not hesitate to send each code you push to peers for review. Even better, if there are engineers from other teams, who have expertise in this area, ask them to review your code.

Some companies implement a two-step code review process when the first review is performed by a teammate, familiar with specifics of a product, while the second review is performed by an expert in language and tools used in the framework.

Follow standards

Most companies or teams have more or less formalized design guide that describes architecture and code style that is supposed to be used. Following same standards for test automation framework makes easier for other engineers to use it in future.

Find the right balance between `fast` and `correct` solution

This is one of the holy war topics in programming — what comes first: code quality or performance, readability or code size. Finding the right balance is absolutely crucial.

It’s OK to use an ugly code for proof of concept when you need to build a feature that works. But keep in mind — if you want the framework to be reusable and maintainable, it’s absolutely necessary to revise these solutions later and refactor them. Add a TODO note, or create a ticket in bug tracking system to improve code once it’s a right time.

Make sure to not postpone refactoring for too long though. The ugly code adds technical debt, which grows exponentially when new features are added to the framework.

Test and debug against correct environments

This is often forgotten by less experienced engineers. A person works on a feature that works great on a local machine, but once run in a cloud or against the real environment — it fails. Localhost allows easy debug but not always covers such issues as:

• Environment performance
• Network issues
• Authorization
• Test data variations

Keep use cases and scenarios in mind

Often, when developing a feature for a framework, engineer thinks about a specific use case for it. While it’s completely reasonable approach, it’s also important to do some analysis and think about other scenarios that can use the feature.

Here are some examples:

• API framework: do not hardcode validation of response code in handling method, instead pass the code as param
• Frontend framework: if scenario requires clicking on the latest search result on the page — make sure to use :last-of-type CSS selector instead of :nth-of-type(20). Otherwise, the test will fail when there are less than 20 results.

Most important part of this advice — do not rely only on yourself to define scenarios. Refer to requirements doc, ask product owner or another engineer, get QAs involved to figure out typical scenarios that need to be handled by the framework.

Follow best practices

One of the reasons to use popular tools is that there are lots of guides describing best practices in designing test automation framework with those tools. There’s a reason why these guides exist — they address typical issues which often lead to flakiness and poor code maintainability. It’s absolutely crucial to follow these guidelines.

Split design and implementation

Purpose of the framework is to simplify actions that may be performed more than once in tests. Instead of writing tests with hardcoded actions, use DRY approach: write tests in BDD format and move repeatable actions to a separate class/library. Now, if an application under test changes in future, you’ll need to modify only one function.

It also simplifies the creation of new tests for other engineers — instead of copying code from the test, they’ll just need to call a function which is expected to be written with an understanding of different use cases.

Follow this pattern:

Tests define what actions will be performed, framework defines how they will be performed

Depending on what tools you use, there might be different patters and tools to implement this approach. Use Page Object Model Selenium-based tests, and implement helpers for API frameworks.

Store test data separately

While having test data hardcoded in test may improve code readability, it should be stored separately if it’s widely reused in framework.

Many tools allow configuring separate test data files which can be easily switched for different environments.

Support debugging

Debugging is one of cornerstones when it comes to using framework by multiple engineers. Good framework support logging, store screenshots / videos of failing and allow running tests in debug mode with as many details exposed as possible.

Provide documentation

Proper documentation is a key to maintainability and reusability of framework. Use JSDoc/JavaDoc for functions that may be called externally. Create a README.md with instructions how to run tests using a framework, debug new tests and analyze issues. If more detailed instructions required — create a Wiki page.

Summary

A good framework is:

• Reusable
• Maintainable
• Reliable
• Simplifies creation and execution of tests
• Aligned to business needs

In order to achieve a high quality of framework it’s important to go through all phases of development, including planning, design and implementation.

Right choice of tools may be a key moment in planning phase, defining how good the framework will be.

Teamwork, collaboration and use of best practices significantly decreases amount of efforts required to build a framework and allows to avoid typical issues.

Good framework provides guidelines how to write new tests, standardizes, and simplifies this process. It’s integrated into software development infrastructure and follows its processes.

In next articles we’ll go through specifics of building E2E, UI and API test automation frameworks.

Please share in comments if you thinks some critical points are missing in this post.