Anatomy of a Test Plan

Some organizations do not proceed with a project until they have an agreed-upon test plan. Some organizations write them along the way. And some organizations never get around to thinking about them at all. To start, let’s talk about what I picture when I say “Test Plan”:

• A handshake, a conversation, a nod
• A whiteboard picture
• A 40-page word document with a detailed list of steps that someone executes manually
• 50-row spreadsheet
• A Cucumber feature file
• An email that lists out a set of high-level test cases
• A set of agile stories marked “QA”

In my opinion, all of the above are perfectly reasonable ways to write a “test plan,” but that process only comes after one already knows what a “test plan” is and how important it is to building reliable software. As the Director for Quality Assurance at Conductor, it is my job to make sure we are building reliable software that meets customer expectations.

In this post, I want to address the most basic thing anyone can do to help the quality of the software they are building. It is all about writing a test plan. That process can take as little as 10 minutes and as long as you want it to. It is all about asking the right questions.

Let’s start with the basics.

What is a “Test Plan”?

A test plan in my book is a codified “What If” scenario analysis targeted at the feature or product at hand.

When building a new feature, good engineering teams allocate time to think thoroughly about how a feature will be built — which components will get re-used, which ones will get augmented, what new ones will be written up and how they will be shared. This is then written out in a class diagram, a design document, or something analogous.

A test plan is an augmentation of that activity. It is the actual allocation of time and effort to ask “What could go wrong and how does/should the product/feature react when it does go wrong — because it inevitably will?”

A test plan is the actual allocation of time and effort to ask “What could go wrong and how does/should the product/feature react when it does go wrong — because it inevitably will?”

Answering this question sometimes feels like identifying and containing infinity. The answer to this question is a delicate dance between some serious thought experiments, and anchored by extrapolated and real experiences. As is true with other designs, the more experience one has, the better the dance will feel. For example, if you have ever been burnt by accidentally deleting critical data, you can be guaranteed your future test plans will include checking data access security and having disaster recovery conversations.

Why write a test plan?

Building good quality software costs money. Every extra test costs either story points or actual money (for manual testing). As engineers, we have the flexibility to decide how much risk we are willing to take on with our features, how much time we are willing to spend up front avoiding a problem, and how much time we are willing to spend fixing a problem later.

Good quality is not an end point; it is a continuum. Writing a test plan is about collectively agreeing to take a risk. Sometimes you will want to take on a lot of risk and sometimes you want to take on very little risk. A test plan identifies a list of risks, and based on that list, enables the team to identify what level of risk they want to take on. Without this analysis and thought, it is not possible to allocate the right level of resources to the right level of problems.

Writing a test plan is all about collectively agreeing to take a risk. Sometimes you will want to take on a lot of risk and sometimes you want to take on very little risk. A test plan identifies a list of risks, and based on that list, enables the team to identify what level of risk they want to take on.

Testing and test planning in general isn’t the first priority if you are just getting your product or app off the ground, and it shouldn’t be. You have to know that you are building the right thing before you build it right. In the early days of a product, verifying that the login for your website works is most likely non-negotiable in terms of risk. However, verifying that there are no extra spaces between “O” and “K” in the “OK” button is probably negotiable on your list of risks.

The purpose of creating a test plan is to get you to think about the risks to a feature’s usefulness in advance, and to help you get to consensus on what level is right for you based on the cost of writing the tests, the project priorities, the ship dates — or anything else that is important to your organization.

How does one write a test plan? What goes in a test plan?

If you have never written a test plan before, it might seem like a daunting task. The trick is to get to the truth of what is important to your customer as quickly as possible.

For example:

• If you are launching the site for 3 accounts and 10 users, your customers probably expect you to be more hands-on with them. In that case, don’t bother with scale testing at all! You might even just want to run through the tests manually instead of taking the hit for a fully built-out automated testing system.
• If you are changing an underlying component in an established product, ensure that the component has the expected behavior. After that, focus on backward compatibility: ensuring that everything that used to work still works, hopefully by running a set of automated tests.
• If you are adding a new feature to an established product, then your users already have an expectation of quality that they will carry forward from the established part of the product — so make sure the new feature meets customer expectations.
• If you are building something that is business-critical, and someone’s job will depend on the reports your product creates, pay extra attention to the correctness of the data.

If you have never written a test plan before, it might seem like a daunting task. The trick is to get to the truth of what is important to your customer as quickly as possible.

How you codify the test plan for your organization totally depends on what works best for you. At Conductor, we sometimes write “feature files” for Cucumber (our test framework, you can read more about in a previous blog post: Cucumber as a Testing Language), we sometimes write an email, sometimes we write actual stories in Jira, and sometimes the test plan is a spreadsheet.

It is not the format that matters; it is the process and the agreement on where the team wants to draw the risk tolerance line for their feature that matter the most. So, if you want to take control of your risks, calculate the risk to your feature or project by writing and discussing a test plan.

Test Plan Cheat Sheet

I want to leave you all with a cheat sheet to getting to a test plan. This is a list of questions I have found useful to mull over when trying to figure out the risk to a feature or project.

It is highly unlikely that all of these questions will be relevant to all projects at all stages of maturity, but asking these questions will help form a good picture of the possibilities, and give you a sense of where the team instinctively draws the risk tolerance line for themselves.

1. How will this feature be sold? This is to guarantee that we deliver what we are being paid to by our customers. We don’t want to promise the customer a scale of service that can’t be tested or supported.
2. Happy Path:
3. How do we want our customers to use this feature?
4. What is the one (or two, or three) specific thing(s) that must work, or else this feature is not worth shipping?
5. Positive/Negative/Boundary Conditions:
6. Where are all the places a customer can deviate?
7. Are all the inputs tightly controlled?
8. Is this change going to break existing tests? Is this expected?
9. Performance:
10. What should customers experience in terms of application responsiveness?
11. What dimensions are important to measure performance? (Throughput? Page load speed? “Chattiness” of the application?)
12. Is performance affected by geographic location of the application’s infrastructure? By the user’s location? If so, what is the acceptable latency for interactions between the two?
13. Will building this feature require changes to any core components that will have ripple effects across other areas? If so, what are the expected effects? Do we have a way to measure them?
14. Scale:
15. What are all the potential places for a pipeline to break?
16. What happens when each link in a pipeline breaks?
17. What if it is not broken but is just really slow?
18. Do we have an application-level monitor to indicate trouble might be on its way? Can we write an automated recovery step?
19. How many concurrent users are we expecting at peak? What is the SLA for performance/availability at that peak level?
20. What load is this going to add to any databases involved? To any other critical pieces of infrastructure?
21. Does this need a long-haul test? (I.e., a load test that hits various portions of the application in a random sequence and in an infinite loop. It is only bounded by time and crashes. The goal is to identify memory leaks and other issues with accumulated state that are too small to be detected in a normal full test pass.)
22. What degradation to total capacity do we want the application to tolerate? When should the alarm bells go off if a capacity limit (concurrent connections to the database, database connection timeouts, database load, front end UI load balancing thresholds) is exceeded?
23. Security:
24. Who uses this?
25. How do we know they are who they are?
26. How do we know they are who they are still?
27. What could someone do if they broke through security?
28. Does it pose a legal risk to us?
29. Is one tenant’s data appropriately walled off from another tenant’s data?
30. In what places does the application talk to untrusted data? Do we have enough validation of the untrusted data in place? Have we tested it? Can we test it? Should we hire an external team to do security penetration testing?
31. In what places does the application talk to untrusted code? (Do you have any analytics plug-ins for your website?) For our purposes, anything we didn’t write ourselves is considered untrusted, because that third-party code can affect our site, but we don’t control that code or the process of shipping it. In these cases, do we have an escape hatch such as turning off the feature on our end?
32. Globalization:
33. Are we selling our product outside of the US? In which countries? In what languages? Do we have any test cases that target this category of users/use cases/tenants?
34. Does the application have a table that has characters that are English? Does sorting/filtering work correctly for this table?
35. What languages should be at the top of the list to verify?
36. Is all the text in the application encoded as Unicode?
37. Fault Tolerance:
38. How many things can fall apart before this system will buckle?
39. What are all the things that can fall apart? Will there be a graceful decline or a dramatic crash?
40. What is the expected time to recover?
41. Recovery:
42. When things go wrong, what is the recovery path?
43. What is the “go back to known good state” plan?
44. Can it be automated for faster recovery?