Photo by NEOM on Unsplash

Exploratory Testing Quick Start

One of the many topics I’ve found to be important to cover early while working with various teams is Exploratory Testing. Many teams struggle with finding sufficient time for testing activities, and Exploratory Testing offers a great set of techniques that can fit in many team contexts. To help get a quick start with Exploratory Testing, let’s go on a short exploration of many of these powerful techniques.

The Testing is Learning Loop (qeek.co)

How is Exploratory Testing Different from Other Forms of Testing?

There are several aspects of Exploratory Testing which help distinguish it from more traditional forms of testing:

• Helps build broader understanding about the system under test (since it tends to explore some areas that traditional, scripted testing might not address)
• Combines real-time test design with test execution
• Invites greater creativity as part of the testing process, by employing various forms of logical decision-making that can take testing in new directions

This quote from James Lyndsay’s paper “Why Exploration Has a Place in Any Strategy” helps illustrate the value of Exploratory Testing:

An automated test won’t tell you that the system’s slow, unless you tell it to look in advance. It won’t tell you that the window leaves a persistent shadow, that every other record in the database has been trashed, that even the false are returning true, unless it knows where to look, and what to look for. Sure, you may notice a problem as you dig through the reams of data you’ve asked it to gather, but then we’re back to exploratory techniques again.

The Importance of Multiple Testing Approaches

To be clear, many forms of testing, automated and otherwise, are an important — indeed vital — part of any testing strategy. It’s also important to recognize that automated tests tend to be good at exposing certain types of issues, but not all types of issues. And it is in those areas where Exploratory Testing can be particularly important.

For instance, test automation is a great choice for executing regression tests that need to be invoked on a recurring basis. By way of contrast, Exploratory Testing can help find things that team members may not have thought of before they started writing code.

Let’s do a brief walk-through of the exploratory testing universe, based on these topics:

• Test Charters
• Test Heuristics
• Exploratory Testing Wheel

Test Charters

A Test Charter outlines a series of related testing activities, often expressed as a goal or a mission for a set of testing activities. Test Charters can delve into all sorts of areas, from validating happy paths for functional tests, to experimenting with different failure modes, to evaluating performance and scalability considerations.

Test Charters sometimes serve as the front end for something called Session-based Testing, which is one of many forms of Exploratory Testing. To keep this blog post reasonably short, we’ll limit this section to Test Charters.

Note: For more information about Test Charters, Session-Based Testing, and many other approaches to Exploratory Testing, see the marvelous series of books by Janet Gregory and Lisa Crispin, which they describe on their Agile Testing website.

What are Some Examples of Some Areas That Test Charters Might Cover?

Here are some examples of what a Test Charter could cover:

• Explore and analyze the main elements of <product/module>. Produce a test coverage outline.
• Identify and test all claims in the <product/module> <release notes/EULA/other>.
• Define work flows through <product/module> and try each one. The flows should represent realistic scenarios of use, and they should collectively encompass each primary function of the product.
• Evaluate the performance/reliability characteristics of <product/module>. Start with a nominal scenario and scale it up in terms of number of options and factors until performance is significantly degraded.
• Try to corrupt an <object/file> when doing <process/function>. What would tell us whether the <object/file> is corrupted?

Test Charter Format

Elisabeth Hendrickson, in her book Explore It! Reduce Risk and Increase Confidence with Exploratory Testing, describes Test Charters this way:

A good charter is a prompt: it suggests sources of inspiration without dictating precise actions or outcomes.

She goes on to suggest that a simple format such as this one works well for Test Charters:

Explore <target>

With <resources>

To discover <information>

The table below provides some additional examples using Elisabeth Hendrickson’s basic Test Charter template.

Sample Test Charter Topics

Ways to Generate Ideas for Test Charters

There are many ways to generate ideas for charters, such as:

• Use Personas
• Use Tours
• Use Acceptance Criteria
• Use the Definition of Done
• Use Risks

Use Personas

Personas help us understand different types of users and how the ways in which they interact with the software we build may differ. Even if no personas have been defined for a product, there are ways to define personas for testing purposes that can be helpful. Here a few imaginative suggestions that can apply to many different types of software:

• Hacker Hal. Either due to boredom or malicious intent, Hal has fun with causing mayhem, and may use various techniques to try to steal or manipulate data or cause performance to degrade. Many of the heuristics that Elisabeth Hendrickson describes in her Explore It! book (summarized later in this post) can help us anticipate Hacker Hal types of actions.
• Concerned Cathy. Cathy hears a lot in the news about identity theft and various companies that lose control of their customers’ data. As a result, Cathy pays close attention to all aspects of the software related to identity management.
• Clueless Claude. Claude does not understand the software well and may take actions that seem odd or unpredictable, which may result in Claude opening a support ticket due to lack of understanding, or because Claude has done something that may require outside assistance.

As an example, suppose that a Test Charter focuses on what Concerned Cathy might be most interested in as a user. Thus it might cover areas such as:

• Search for users
• Authenticate users’ identity
• Tampering with user data (and using log files or other monitoring mechanisms to detect such tampering attempts)

Use Tours

As the name implies, using tours in a Test Charter context means we think in terms of different journeys users could take through an application. As such, we think of a tour we can take based on a particular theme, navigate through the system based on that theme, and document findings. Using tours can also help surface a collection of ideas that might have been thought of, which can be explored one at a time.

Below are some examples of tours, suggested by various practitioners.

James Whittaker Tours

• Money Tour: Test major features that have not already been tested in other scenarios (it’s called “money” because the focus is on significant features that might constitute a significant departure from previous functionality)
• Guidebook Tour: Test the software’s ability to deliver its advertised functionality
• Landmark Tour: Choose a set of “landmarks” (important user decision or user action points in the system), decide on an ordering for them, and then explore the application going from landmark to landmark until you’ve visited all of them in your list.
• Garbage Collector’s Tour: Based on a particular goal (for example, use all menu items, expose all error messages, use all dialog boxes), visit each one in the list via the shortest path possible.
• Rained-out Tour: Explore various ways to interrupt or cancel operations in the software, like starting and stopping tasks, hitting Cancel, using the browser Back button, etc.
• Obsessive-compulsive Tour: Perform tasks multiple times, and check for differing results.
• Back Alley Tour: Test areas of the system that have little test coverage or that users only work with on an occasional basis.
• All-Nighter Tour: Leave the application/system open overnight (or some similar long time window) and see what happens.
• Collector’s Tour: Document every output you see as you run through scenarios, and see if you can create more scenarios based on the outputs collected.

Note: For more information about Jame’s Whittaker’s tours, see his book Exploratory Software Testing: Tips, Tricks, Tours, and Techniques to Guide Test Design.

James Bach Tours

• Documentation Tour: Look in the online help or other documentation, find some instructions about how to perform a particular activity, perform the actions as described, and see whether the behavior matches what is in the documentation.
• Sample Data Tour: Employ any sample data that you have (or create some sample data), and run tests using that data.
• Variability Tour: Tour the system with a mind toward exploring different logic paths/logic loops, exploring as many dimensions as possible.
• Complexity Tour: Tour the system looking for areas where the highest levels of complexity exist, such as retrieval or manipulation of large or multi- faceted data sets.
• Continuous Use Tour: While testing, do not follow the instructions in terms of how long the system is intended to be used; leave windows and files open, see if disk and memory usage increase.

For more about James Bach’s observations of Exploratory Testing, see his Software Testing for Serious People blog.

Michael Kelly Tours

Michael Kelly’s tours follow the mnemonic FCC CUTS VIDS:

• Feature Tour: Move through the system and get familiar with all the controls and features you come across
• Complexity Tour: Find the five most complex things about the system and explore those areas.
• Claims Tour: Find all the information in the system which tells you what the it does, including artifacts like End User License Agreements, and execute tests to explore those claims.
• Configuration Tour: Attempt to find all the ways you can change settings in the system in a way that the system retains those settings.
• User Tour: Imagine five users for the product and the information they would want from the product or the major features they would be interested in.
• Testability Tour: Find all the features you can use as testability features and/or identify tools you have available that you can use to help in your testing.
• Scenario Tour: Imagine five realistic scenarios for how the users identified in the User Tour would use this system.
• Variability Tour: Look for things you can change in the system (even if they are not supposed to be changed), and then try to change them. Interoperability Tour: What does the system interact with? Explore based on those interactions.
• Data Tour: Identify the major data elements and data operations associated with the system.
• Structure Tour: Find everything you can about what comprises the physical product (code, interfaces, hardware, configuration files, etc.) and explore based on what you see in those artifacts.

For more insights from Michael Kelly, see his article Taking a Tour Through Test Country: A Guide to Tours to Take on Your Next Test Project.

Use Acceptance Criteria

The Acceptance Criteria associated with particular user stories can potentially be combined in interesting ways to inform areas where additional testing could be helpful.

Consider for instance this set of Acceptance Criteria, which may have originated from multiple user stories that are related, but have been split to reduce the size of each one:

• A user cannot submit a form without completing all the mandatory fields
• Information from the form is stored in the registrations database
• Protection against spam is working after submitting the form
• Users can pay by credit card when submitting the form
• An acknowledgment email is sent to the user after submitting the form

One or more Test Charters could explore various behaviors related to form submission, above and beyond the behaviors described in the aforementioned set of Acceptance Criteria.

Use the Definition of Done

A Definition of Done (DoD) often contains references to completion of various forms of testing, and also various characteristics of a system, such as availability, performance during peak usage periods, and security, to name a few.

Consider for instance the DoD below:

Unit Tests

When writing unit tests, be sure to cover classes and methods that:

• Have few if any dependencies
• Do not interact with resources like a database, a file system, or third party systems
• Have underlying functionality that cannot be properly covered by an integration test

Integration Tests

When writing integration tests, consider covering classes and methods that:

• Cannot be covered with a black box unit test
• Have many dependencies (for instance, for cases where a unit test would require complex mocking)
• Interact with external resources (database, file system, third party systems)

Load Tests

When designing and executing load tests, the following performance expectations need to be met:

• System performance for <number of users> should be within <performance targets>
• System performance when <abnormal condition exists> should be within <performance targets>

Many Test Charters could be created by using the DoD above (or any DoD) as a starting point.

Use Risks

Teams often identify various types of risks as they design, build, and deploy products. It can be helpful to create Test Charters that explore in various areas related to those risks.

Below are some examples.

• <Deliverable> cannot be completed until approval is received from <approver> (for instance, Legal approval)
• <Activity> (for instance, static code scan) results in <set of findings>, some of which must be addressed before release
• <Deliverable> requires completion of <activity> by <External entity>

Test Heuristics

Elisabeth Hendrickson’s book Explore It! Reduce Risk and Increase Confidence with Exploratory Testing, contains a set of test heuristics in one of the appendices. Below are some examples:

Beginning, Middle, End

Vary the position of an element. You could vary where a given element appears in a sequence, or you could act on an element in a given position. Examples of applying this heuristic include the following:

Deleting an item from the beginning, middle, or end of a list

Pasting text at the beginning, middle, or end of a line

Placing a special character at the beginning, middle, or end of a value

Performing an action at the beginning, middle or end of a sequence

Watch for indications that position makes a difference. Also be alert for bad behavior, such as data appended at the end being truncated or data inserted at the beginning or in the middle causing other contents to be truncated.

Change the Model

Take a model of the system you’re testing and translate it into a different representation of the model. Examples:

Change a state diagram with circles and arrows representing the states and transitions into a table showing states as columns, events as rows, and resulting states as data in the cells.

Change a linear outline into a mindmap, or a mindmap into a linear outline.

CRUD

Create. Read. Update. Delete. These are the essential actions you can take on any data element within a system. Use CRUD in conjunction with other heuristics. For example:

With Beginning, Middle, End: Create an element at the beginning of a list, in the middle of a list, or at the end of a list.

With Goldilocks: Update a data element with a value that that is too small, too big, or just right.

With Zero, One, Many: Delete an item with no child records, with one child record, or with many child records.

Follow the Data

Perform a sequence of actions involving data, verifying the data integrity at each step. This is appropriate for any system where users can perform activities on data like entering, searching, reporting, importing, exporting, updating, and viewing. Following the data is particularly useful when combined with other data-related heuristics.

For example:

With data that has special characters, including SQL injection or JavaScript injection attacks: Import it. Search on it. View it. Run reports that include it. Export it.

Create data that violates domain-specific rules and then import it. Sometimes the logic that prevents bad data from being entered directly is missing from batch import utilities.

With Zero, One, Many: Perform data-centric actions such as running reports or exporting data with entities that have no child records, one child record, or many child records.

Goldilocks

Too big, too small, or just right. This can apply to anything that has a valid range of sizes, such as a range of dates, a numerical value, the length of a string or text, or the size of a file.

Behavior to watch for includes the following:

Unhelpful error messages, such as raw exceptions with stack traces or mysterious error codes.

Truncating large data without notice

Failing to save data without notice

Interrupt

Find ways to interrupt key processes, such as logging off, shutting down the machine, rebooting the machine, killing the process from the operating system, disconnecting the machine from the network, forcing the machine into hibernation, letting the session time out, and cancelling the operation.

Watch for misbehavior, such as the following:

Unhelpful error messages, such as raw exceptions with stack traces or mysterious error codes

Inability to start the process (ever) again

Unrecoverable data loss or corruption

Some, None, All

Some of the items in a set, none of them, all of them. Related to 0, 1, Many but for defined sets of items or attributes such as permissions, configuration options, tags, multiselect fields, and checkboxes.

Behavior to watch for includes the following:

None treated the same as All –for example, the system treating a user with no permissions the same as a superuser with all permissions

Problems with count or percentage calculations, including divide-by-zero and off-by-one errors

Display issues, particularly with None or All

Data truncation with All

Starve

Starve the program of any required resource, including CPU, memory, network bandwidth, or disk space. Watch for these responses:

Unrecoverable data loss or corruption

Unhelpful error messages, such as raw exceptions with stack traces or mysterious error codes

Unexpected program or session termination

Zero, One, Many

Zero things, one thing, many things. The things you are counting might include results returned in a search, characters, in a string, bytes in a data stream, descendants in a hierarchy, a number of files, lines within a file, accounts, activities in a log, or any domain-specific object or attribute.

Behavior to watch for includes these issues:

Problems with plurals, such as “0 record found” or “1 records found”

Problems with count or percentage calculations, including divide-by-zero and off-by-one errors

Indications of performance issues as the count of things increases

Exploratory Testing Wheel

In his book Visualization Examples: Toolbox for the Agile Coach, Jimmy Janlen describes close to 100 information radiators that can be helpful to many Agile teams.

One of those is the Exploratory Testing Wheel, an example of which is shown below.

Exploratory Testing Wheel

To use the Exploratory Testing Wheel, let’s say a team has a set of half a dozen Exploratory Test Charters, which each one is represented by one of the green cards in the example above.

• Team members could agree on a cadence for how often to pick and execute a Test Charter from the wheel (for instance, one Test Charter per week).
• When a Test Charter from the wheel has been completed, indicate that it’s done on the wheel (in the example, an “X” is placed next to it).
• When all of the Test Charters on the wheel have been completed, restart the wheel, possibly changing the set of Test Charters