Charters: Exploratory Testing… and Beyond

I’ve amassed quite an arsenal of tools and practices for Agile Software Development and Lean Product Management. Of these, my go-to techniques for maintaining a high level of quality are Exploratory Testing and Charters.

Exploratory Testing

Exploratory Testing is the art of deciding on what to test next. These decisions are based on your past learnings from this application, as well as other applications you’ve worked on in the past. It can be based on known weaknesses in the code base, prior bugs, and which features and integration points were most recently implemented. Charters are a way of harnessing your testing intuition into a format that’s not just a free-for-all. Charters set the purpose of a given exploration, as well as the scope.

I first started to dabble in exploratory testing after attending a talk by Jon Bach at STAREAST Software Testing Conference in 2010. Years later, I was delighted to read fellow Pivot Elisabeth Hendrickson’s book Explore It!, which defines a Gherkin-like story format to help specify the purpose and scope of a Charter:

Explore <target> 
With <resources>
To discover <information>

With this template, product teams are empowered to treat their Exploratory Testing Charters as first-class stories. Charters can be prioritized in the backlog amongst the Features, Chores and Bugs according to the value they provide. Let’s break down the template to give you a better idea of what a Charter might look like:

Explore <target>: The target might be a specific feature, an area of code, a new package or gem that was just introduced into the system, an integration point, etc.

With <resources>: This is how to define the tools the tester should use, such as a packet sniffer or a load generator. It could also define the characteristics of the target input data, such as uncommon character encodings or very long strings. The resources section could also be used to define constraints or conditions to avoid over-testing, diving too deep, or to time-box the endeavor.

To discover <information>: This is the purpose of the Charter. Do you want to find out if something works? Is it reliable? Is it secure? Does it perform well? Is it user-friendly? Do the workflows make sense?

The output of a Charter is an experience report of the information discovered along the way. How you capture that output is up to you, but I prefer to add everything to the story itself. The report does not need to be formal or structured, but it should include both the primary information you were looking for in the first place, as well as any incidental learnings. Any bugs that were discovered should be mentioned in the report, but they should also be reported as usual.

… and Beyond

Eventually, some of the Charters in my backlog seemed to take on a purpose that wasn’t quite in the realm of exploratory testing. We started using the Charter format to prioritize different varieties of exploration.

During our weekly Iteration Planning Meeting, the whole team gets together and reviews each upcoming story to make sure there’s a shared understanding of the purpose and the acceptance criteria. The engineers estimate the relative complexity, as compared to other stories recently completed. Sometimes, the engineering team simply doesn’t have enough information to even warrant a guess as to how complex a story will be. Teams often create a Spike to find out more, but I started using Charters instead.

When a team works on a Spike, all of the knowledge learned typically ends up only in the minds of the engineers who worked on it. There’s no expected output other than a verbal, “here’s how we can do it, and I estimate it’s a <size> with respect to complexity”. All of the interesting bits might not have been shared, such as how the decision was made, what other methods were tried, and lessons learned.

When a Charter is used to explore this information instead, the experience report would include all of the incidentals learned. The Charter provides a structure for how to go about finding the information needed, such as writing some throw-away code, reading reviews about different third-party services, or asking experts in the office. Using a Charter in this manner certainly isn’t exploratory testing, but it still feels a lot like exploring.

I now use Charters so frequently in my backlog that I helped build Charters for Pivotal Tracker, a Chrome Extension that gives any story with a “charter” label a shiny new “🚀” icon. Many thanks and much credit to David Edwards and Daphne Lin!