“Manual” Regression in Agile — A focused approach

8 min readDec 7, 2017

As promised in my article on The Testing Role in Agile I wanted to dig a little deeper into approaching “manual” regression testing in an Agile environment. I also set some expectations around expected skill levels and test cases after receiving some feedback on the original post.

There is an assumption in Agile that all regression should be automated. The reality is there may be some things that can’t or should’t be automated. There is always some element of manual regression needed in supporting most products. That effort can be difficult given the need for fast feedback and the pressures of Continuous Integration or Continuous Deployment. This post outlines how you can target your testing and include the whole team in the effort. It’s important to have a comprehensive strategy for your regression testing that should take both manual and automation into account. I am going to focus on the manual side for this post but I will note that this is a good complement to automation. The goal of this strategy is to set clear expectations and focus on the areas of high risk.

This strategy also breaks my rule of having all project related information in one tool such as Jira. The best tool I have found for it is a Google Docs Spreadsheet. It makes it easy to share and all updates are seen in real time which is necessary to get the most out of this approach. There may be a better tool that is easier to integrate into your project tracking tool but this post will assume Google Docs. Please share in comments if you have any other tool suggestions.

Getting Started

While it is important for this to be a group effort the person fulfilling the testing role (which I will refer to as QA moving forward) should be the one driving the effort. Driving doesn’t mean that someone does everything, it just means they are responsible with making sure you get somewhere safely. Creating a regression strategy is part of creating an environment in which quality can occur which is an important responsibility of those in QA.

I’ll list out the steps for QA and then break them down:

Create a rough start to the checklist
Document your intended strategy at a high level including definitions (I am hoping this article will make this easier)
Set up a meeting with stakeholders and set the expectation this is a working meeting with an actionable document as a takeaway
Review goals, definitions, and update checklist as a group
Share document to all project stakeholders
Execute as agreed
Update and repeat steps as needed by project

The Checklist

This process does not create step by step scenarios for testers to follow. Rather it needs to be as high level as possible. This allows the document to be easy to update and maintained by QA. It assumes those using the checklist are familiar with the product and are capable of some level of exploratory testing. Remember that this is for areas of your product that is hard or not valuable to automate. If something needs to be repeated the same way over and over then you should invest the effort into automating it.

In order to have an effective meeting it is helpful to have a place to start. That way the team isn’t working from a blank slate. Here is an example starting checklist:

You can customize and organize the checklist in a number of ways but this is a good place to start. Keep in mind that this will be updated during the group meeting so doesn’t need to be comprehensive as a first draft.

Functional Areas — High level areas that make sense for your project. They could be pages, modules, templates, or any logical piece of functionality

Priority — As the goal of this process is to focus testing prioritizing each area is necessary

Status — This will become a living document that will also function as real time reporting so a status for each area under test is also necessary. This also provides some historic traceability on potentially problematic areas

Tester — This document is meant to be used by the team and in some cases everyone will be using it at the same time. Marking names prevents people overlapping so the team can complete the checklist faster

Jira # — I used Jira here but this can be any tool. The assumption is bugs will be uncovered during this process and it’s helpful to link the bug created to the spreadsheet. It’s a flaw of using a seperate tool.

Notes — Not really required but sometimes useful to document some additional information

Definitions

Prior to the meeting it’s helpful to set some expectations so documenting some terms may be helpful. You can use what I have here or customize to your own needs. If you are customizing it’s important to keep in mind a few factors.

If everything is important then nothing is. This strategy isn’t going to work if every functional area is a P1. It’s better to accept the reality that time is a factor. It’s likely you won’t have time to test everything you would like to. It’s better to run out of time while testing less risky areas then to try to test everything and miss a Blocker. Be strict and realistic with timeboxing your efforts.

As helpful as it is to list out the high risk areas it’s also important to list out areas that you won’t be testing. This decision will be a made as a group so everyone is agreeing to it. It is also documented so those outside of the group can also be aware. However, even if you don’t plan on testing these areas it’s still helpful to know how long it will take to execute each priority in case you need to do a full pass.

Keep in mind that priorities can and should change as your product develops. If you aren’t sure of a priority you can choose a place to start and change it later. This document is lightweight in order to make such adjustments easy.

Timeboxed

The team will need to decide how much time every sprint to dedicate to manual regression. This might be 4 hours or it might be 2 days. It’s important to decide this in order to plan future sprints. This way during Sprint Planning sessions you will factor in this time and only take on Stories that can be completed in the remaining time. Keep in mind that the time for fixing any bugs you find should also be factored in the time you choose to dedicate to regression.

Priorities

Will be completed every Sprint
Areas of high technical risk
Areas of high business value
Not covered by any automation
Timeboxed

Will be completed as time allows
Areas of some technical risk
Areas of some business value
May have overlap with automation
Not timeboxed
Total effort known

May never be completed
Areas of low technical risk
Areas of low business value
Covered by automation
Total effort known

The Meeting

At a minimum I would say this meeting will need to be attended by the QA Lead, Product Owner, and Development Lead. You need to be able to understand business need, technical risk and test efforts in order to create a functional checklist. However, it’s preferred to have the entire team present as well as other stakeholders such as Project Managers. This creates more of a sense of team level ownership over the regression process. This will likely be an hour long meeting but may take longer depending on the complexity of your product. The goal is to walk out of the meeting with an actionable checklist and a shared agreement on using it.

QA should present the draft version of the checklist and review the definitions needed in order to update it. An agreement should then be made on how much time to dedicate to regression testing each Sprint. The document should be shared among those attending and updated in the meeting as decisions are made.

Your checklist should look something like this at the end of the meeting:

Using the Checklist

So now that there was a group discussion and agreement you have a master checklist that you can use and the time to implement it. It’s important to share the link to the document to all stakeholders since this will not be tracked inside of your project tool. This is a great answer to “How is regression testing going?”. Anyone with the link will have that information in real time leaving more time for testing rather than updating stakeholders.

To get started make a copy of the “Master” sheet that was created during the set up meeting. A new sheet copy should be created at each regression pass. This helps focus the testing on what needs to be completed for this pass only. You can remove areas on the sheet if you aren’t going to be testing them if you wish. Having a new tab for each pass also allows you to have an historic record so you know if an area continuously problematic and may need to refactored.

Regression can be a team testing effort with everyone helping with regression during the time you have timeboxed. Each team member helping can claim a section to focus on. It’s suggested that those in QA focus on the P1 areas as they should have the deepest exploratory testing skill. However, other team members, including developers, can also assist in the effort.

Each person testing should update the status and results frequently. This provides updates in realtime. Discussion can be had on the bugs uncovered and if they need to be addressed immediately or added to the backlog.

Keep it updated

As new features are developed or changed than the master sheet should also be updated. It shouldn’t be necessary to have a group meeting with each update as the team should now be familiar with the goals and checklist. The QA Lead should initiate feedback from Product and Development as needed.

In Summary

I’ve had the opportunity to experiment quite a bit with different approaches to regression in Agile. This method was the most successful when properly implemented and can work both with and without automation. The complaint I got most often around regression was when teams attempted to test everything and ran out of time. This approach makes the time needed and focuses on what is the most important areas to test. It also focuses on the whole team so individual testers do not become bottlenecks.

Using spreadsheets may seem like an unsophisticated archaic way of solving the problem. It’s the best tool I’ve found to solve this particular issue. You may also look into using mindmaps and follow the same process.

However, with a spreadsheet you are able to create some metrics if you need to and it’s a little easier to have the historic record all in one place with tabs. Mindmaps are a useful tool and for some teams can be a great solution.

What other tools would work with this process?