Black Box Testing Techniques
In previous article we discussed that there are different classifications of test design techniques, and one of them is based on the access to code and application architecture, including Black Box and White Box Testing techniques.
Let’s talk about Black Box Testing techniques in details.
Table of Contents:
Equivalence Class Testing
Boundary Value Testing
Decision Table Testing
Pairwise Testing
State-Transition Testing
Domain Analysis Testing
Use Case Testing
Equivalence Class Testing (also called Equivalence Partitioning)
Equivalence Class Testing is a technique used to reduce the number of test cases to a manageable size while still maintaining reasonable coverage.
This technique is used intuitively by almost all testers, even though they may not be aware of it as a formal test design method.
ISTQB Definition of Equivalence Partitioning
A black-box test technique in which test conditions are equivalence partitions exercised by one representative member of each partition.
How to Implement
The steps for using equivalence class testing are simple:
First, identify the equivalence classes.
Second, create a test case for each equivalence class.
A group of tests forms an equivalence class if:
- They all test the same thing.
- If one test catches a bug, the others probably will too.
- If one test doesn’t catch a bug, the others probably won’t either.
Testers could create additional test cases for each equivalence class if they have the time and money. Additional test cases may add more confidence, but they rarely discover defects the first doesn’t find.
Example
Consider the ‘Phone Number’ field. The requirements for this field are:
- two to fifteen characters inclusive
- only numeric characters are allowed (0, 1, …, 8, 9)
We got three equivalence classes:
[0, 1] — invalid length
[2, 15] — valid length
[16, ∞] — invalid length
Valid inputs are {112, 555123, etc.}
Invalid inputs are {1, 555–123, 555(123)455, etc.}
So, with Equivalence Class Testing, testers identify these classes that produce the same results and only test for one value within that class.
Boundary Value Testing
While Equivalence Class Testing is useful, its greatest contribution is to lead to Boundary Value Testing.
Boundary Value Analysis an instrumental testing technique based on equivalence classes, which makes it possible to identify specific values of the parameters under study that belong to the boundaries of equivalence classes.
It focuses on the boundaries because that is where so many defects hide.
Experienced testers have encountered this situation many times.
Unexperienced testers may have an intuitive feel that mistakes will occur most often at the boundaries.
ISTQB Definition of Boundary Value Analysis
A black-box test technique in which test cases are designed based on boundary values (a minimum or maximum value of an ordered equivalence partition).
How to Implement
Create test cases for each boundary value by choosing:
- one point on the boundary,
- one point just below the boundary,
- and one point just above the boundary.
“Below” and “above” are relative terms and depend on the data value’s units.
Example
Consider the ‘Phone Number’ field. The requirements for this field are:
- two to fifteen characters inclusive
- only numeric characters are allowed (0, 1, …, 8, 9)
A set of boundary value test cases for the length attribute would be {0, 1, 2, 15, 16} numeric characters.
Decision Table Testing
Decision Table Testing can be used whenever the system must implement complex business rules.
Conditions represent various input conditions.
Actions are the processes that should be executed depending on the various combinations of input conditions.
Each rule defines a unique combination of conditions that result in the execution (“firing”) of the actions associated with that rule.
Choosing test cases is simple — each rule (vertical column) becomes a test case. The Conditions specify the inputs and the Actions specify the expected results.
ISTQB Definition of Decision Table Testing
A black-box test technique in which test cases are designed to exercise the combinations of conditions and the resulting actions shown in a decision table.
How to Implement
Decision table testing involves enumerating each combination of inputs and its expected outcomes and developing a test case to validate each combination.
Create at least one test case for each rule.
If the rule’s conditions are binary, a single test for each combination is probably sufficient.
On the other hand, if a condition is a range of values, consider testing at both the low and high end of the range.
Example
Let’s look at a Credit card example.
If you are a new customer opening a credit card account, you will get a 15% discount on all your purchases today.
If you are an existing customer and you hold a loyalty card, you get a 10% discount.
If you have a coupon, you can get 20% off today (but it can’t be used with the ‘new customer’ discount).
Discount amounts are added, if applicable. This is shown in the table below.
Pairwise Testing
When the number of combinations to test is very large, do not to attempt to test all combinations for all the values for all the variables, but test all pairs of variables. This significantly reduces the number of tests that must be created and run.
Studies suggest that most defects are either single-mode defects (the function under test simply does not work) or double-mode defects (the pairing of this function/module with that function/module fails). Pairwise testing defines a minimal subset that guides testers to test for all single-mode and double-mode defects. The success of this technique on many projects, both documented and undocumented, is a great motivation for its use.
ISTQB Definition of Pairwise Testing
A black-box test technique in which test cases are designed to exercise pairs of parameter-value pairs.
How to Implement
Two different techniques are used to identify all the pairs for creating test cases:
1. Orthogonal Arrays
2. Allpairs Algorithm
Using Orthogonal Arrays
An orthogonal array is a two-dimensional array of numbers that has this interesting property — choose any two columns in the array, all the combinations will occur in every column pair.
The process of using orthogonal arrays to select pairwise subsets for testing is:
1. Identify the variables.
2. Determine the number of choices for each variable.
3. Locate an orthogonal array which has a column for each variable and values within the columns that correspond to the choices for each variable.
4. Map the test problem onto the orthogonal array.
5. Construct the test cases.
Using Allpairs Algorithm
Using orthogonal arrays is one way to identify all the pairs. A second way is to use an algorithm that generates the pairs directly without resorting to an “external” device like an orthogonal array.
There are special programs (e.g. Allpairs), that take a table of all the variables and their values and generate the list of all pairs test cases.
Note that the combinations chosen by the Orthogonal Array method may not be the same as chosen by Allpairs. It does not matter. What does matter is that all of the pair combinations of parameters are chosen. Those are the combinations we want to test.
Example
Consider there is an online book store. Books can be purchased on a website or via a mobile app, and it is expected to work for all browsers, all operation systems, and both Android and iOS.
The total number of all combinations becomes very large. So instead of testing all combinations for all the values for all the variables, consider testing all pairs of variables using Pairwise Testing.
State-Transition Testing
An application may be designed to change state under certain conditions, such as locking a user’s account after a certain number of failed authentication attempts.
State-Transition diagrams direct our testing efforts by identifying the states, events, actions, and transitions that should be tested. Together, these define how a system interacts with the outside world, the events it processes, and the valid and invalid order of these events.
State-Transition diagrams are not applicable when the system has no state or does not need to respond to real-time events from outside of the system.
ISTQB Definition of State Transition Testing
A black-box test technique in which test cases are designed to exercise elements of a state transition model.
How to Implement
Information in the state-transition diagrams can easily be used to create test cases.
Four different levels of coverage can be defined, so create test cases verifying that:
1. all states are “visited”
2. all events are triggered
3. all paths are executed (most preferred because of its level of coverage, but sometimes may not be feasible)
4. all transitions are exercised (provides a good level of coverage without generating large numbers of tests)
The generally recommended level of testing using state-transition diagrams is to create a set of test cases such that all transitions are exercised at least once under test. In high-risk systems, you may want to create even more test cases, approaching all paths if possible.
Note that in any given state, one event can cause only one action, but that the same event — from a different state — may cause a different action and a different end state.
Example
An example is entering a Personal Identity Number (PIN) to a bank account. A state diagram for PIN entry is imaged below. It can be expanded more, but this representations is quite good to demonstrate the concept.
The states are shown as rectangles, the transitions as lines with arrows and the events as the text near the transitions.
The state diagram shows seven states but only four possible events (Card Inserted, Enter PIN, PIN OK and PIN not OK).
Domain Analysis Testing
Sometimes we need to test multiple variables simultaneously. There are two reasons to consider this:
- We rarely will have time to create test cases for every variable in our systems. There are simply too many.
- Often variables interact. When the value of one variable constrains the acceptable values of another, certain defects cannot be discovered by testing them individually.
Domain analysis is a technique that can be used to identify efficient and effective test cases when multiple variables (such as input fields) should be tested together either for efficiency or because of a logical interaction.
It builds on and generalizes Equivalence Class and Boundary Value Testing to n simultaneous dimensions. Like those techniques, we are searching for situations where the boundary has been implemented incorrectly.
Definition of Domain analysis
A black box test design technique that is used to identify efficient and effective test cases when multiple variables can or should be tested together.
How to Implement
The domain analysis process guides us in choosing efficient and effective test cases.
First, a number of definitions:
- An on point is a value that lies on a boundary.
- An off point is a value that does not lie on a boundary.
- An in point is a value that satisfies all the boundary conditions but does not lie on a boundary.
- An out point is a value that does not satisfy any boundary condition.
Having defined these points, the 1 x 1 (“one-by-one”) domain analysis technique instructs to choose these test cases:
- For each relational condition (≥, >, ≤, or <) we choose one on point and one off point.
- For each strict equality condition (=) we choose one on point and two off points, one slightly less than the conditional value and one slightly greater than the value.
Examples of on, off, in, and out points for both closed and open boundaries.
While this technique is best suited to numeric values, it can be generalized to Booleans, strings, enumerations, etc.
Use Case Testing
Use Case Testing is a technique that helps identify test cases that exercise the whole system on a transaction by transaction basis from start to finish.
A use case is a description of a particular use of the system by an actor (a user of the system).
Each use case describes the interactions the actor has with the system in order to achieve a specific task (or, at least, produce something of value to the user).
Actors are generally people but they may also be other systems.
Use cases are a sequence of steps that describe the interactions between the actor and the system.
The value of use cases is that they:
- Capture the system’s functional requirements from the user’s perspective; not from a technical perspective, and irrespective of the development paradigm to be used.
- Can be used to actively involve users in the requirements gathering and definition process.
- Provide the basis for identifying a system’s key internal components, structures, databases, and relationships.
- Serve as the foundation for developing test cases at the system and acceptance level.
ISTQB Definition of Use Case Testing
A black-box test technique in which test cases are designed to exercise use case behaviours.
Example
The PIN example that we used for State-Transition Testing could also be defined in terms of Use Cases, as shown in table below.
While creating at least one test case for the main success scenario and at least one for each extension provides some level of test coverage, it is clear that, no matter how much we try, most input combinations will remain untested. Do not be overconfident about the quality of the system at this point.
Conclusions
As you could see, there are various Black Box Testing techniques available, and if you discover one that captivates your interest, you can delve into its details using the sources provided below.
Sources
A Practitioner’s Guide to Software Test Design by Lee Copeland
Foundations of Software Testing: ISTQB Certification by Dorothy Graham
Software Testing. Base Course by Svyatoslav Kulikov
