How Cem Kaner tests software
In the world of software development, Cem Kaner should get the highest recognition for his contribution to software testing (I will describe Kaner’s contributions in another post). In his first book, ‘Testing Computer Software’, Kaner explains how to test a program . Oddly, this is the only description of how someone actually tests software . It’s not easy to put yourself on the line to explain how you think, and be accountable for your thinking process. I have created a shortened version of Kaner’s description in this post. Note that I have shortened sentences in some cases, but in most other cases I have copied his exact text. To really appreciate his work, I strongly recommend reading his book.
If you are a tester this gives a great perspective on the meaning of testing. If you are a blogger, author or leader, it’s worth trying to write a similar account of how you test software.
Note that the first edition of this book was written in 1988. Kaner’s ideas on testing have evolved since then. However, this description of testing hasn’t changed and most likely won’t change in the future. For all testers and developers, being able to test like this is a significant first goal.
The first cycle of testing
You’ve been given the program and the following description of it:
The program is designed to add two numbers, which you enter. Each number should be one or two digits. The program will echo your entries, then print the sum. Press <Enter> after each number. To start the program type ADDER.
Step 1: Start with an obvious and simple test
Familiarize yourself with the program. Is it stable enough to be tested?
Add 2 + 3
Problem reports from the first test
The program returned 5, but….
1. Design error: Nothing shows you what program this is.
4. Coding error: The sum (5) isn’t lined up with the other displayed numbers.
Step 2: Make some notes about what else needs testing
Make notes about what else needs testing. Groups of tests, called test series, e.g., valid inputs will be re-used every time there is a new version of the software.
Valid inputs may be 99 + 99, -99 + -99
Use boundary conditions. 99 and 100 are boundary cases for two digit numbers.
Don’t repeat tests in the same class. 2 + 3 and 3 + 4 can be considered to be the same class.
There are no magic formulas for boundary conditions or test classes.
It is your task to analyze the program from a different point of view than the programmer’s. This will help you find classes, boundary conditions, critical tests, and thus errors that she missed.
Make sure you test all sides of a boundary.
Step 3: Check the valid cases and see what happens
Create a few more series — 1. invalid values 2. numbers you enter and then edit
Step 4: Do some testing ‘on the fly’
When you run out of formally planned tests, keep testing. Run new tests as you think of them…. Trust your instincts. Try any test that feels promising, even if it’s similar to others that have already been run.
This program crashed quickly. At this point, rather than waste time on tests which may become obsolete with the new version, try some exploratory tests. e.g., 100 + 100, <Enter> + <Enter>, 123456 + 0
Hand in your problem reports and perhaps write a summary memo about them.
Step 5: Summarize what you know about the program and its problems
Write down a list of points summarizing your thoughts about the program.
- The communication style of the program is extremely terse.
- The program doesn’t deal with negative numbers. ….
It’s possible that the reason for the errors is that the programmer is trying to make the program really fast or very small. You can only know this by talking to the programmer.
At this point Kaner does a great analysis of how the storage of the sum and the storage of the input could give rise to errors.
The Second cycle of testing
Step 1: Before doing any testing, review the responses to the problem reports carefully to see what needs to be done, and what doesn’t
1. Design issue: No program title onscreen.
Resolution: Won’t be fixed
4. Bug: The sum (5) isn’t lined up with the other displayed numbers
The programmers hasn’t added error handling and numbers from -10 to -99 aren’t handled. You modify the tests to use single digit negative numbers.
While the programmers is busy you can still test the rest of the program. Modify your tests to work with the program in it’s current state. Don’t wait to test your ‘best tests’.
Step 2: Review comments on problems that won’t be fixed. They may suggest further tests
7. Bug Crashes when you enter non-numbers.
Resolution Not a problem. Comment: “Don’t do that.”
A good way to find the worst (best) examples of a bug’s misbehavior is to boil it down to its simplest, barest essentials.
What if the program crashes with characters that people would expect to work? Try to brainstorm such cases. e.g., plus sign, trailing spaces, delete key, backspace. When brainstorming don’t restrict yourself to what makes sense. Create a list which doesn’t miss anything.
Step 3: Pull out your notes from last time, add your new notes to them, and start testing
It’s tempting to start with the complicated, brilliant new test cases you just thought of. Don’t. Start with those drudge tests that confirm that the program can still add 2 and 2 and not get 5. …… Test the basics first.
Try everything in the formal series…..It all works.
You notice that the program says Press Ctrl-C to quit after each addition. Since the programmer had said that the speed of the program is an issue the problem is a bug.
10. Design Error: Writing “Press Ctrl-C to Quit” on the screen after each result wastes a lot of machine time……
You check single byte sums. It all works. ‘Oh, well’.
Don’t stop thinking. Some of your tests will find problems, and the more care you put into crafty thinking, the more you’ll find.
You check invalid characters, the ones that matter and they all crash the program. File a problem report.
What will happen in later cycles of testing
As development progresses, you will create more formal test series, and will follow them each time the program returns to you. Once a few versions of the program have consistently passed every test in a series, you’ll probably use only a few of these tests in later cycles. To be safe, try to rerun every test in what you think is the final cycle. Before that, why run tests that a program can pass?
As the program gets closer to being finished, you’ll use stricter tests. You’d rather run the toughest tests first, but you won’t think of many of them until you’ve tested the program for a while and learned it’s quirks.
Along with using tests to expose new errors, you’ll look for ways to reopen consideration of problems that you’ve been told won’t be fixed, but that you feel are important. You will not win every battle, nor should that be your goal. Attempts to fix a program can do much more harm than good. Near the release date, some problems are best left alone. Your objective is to make sure that a problem’s severity is clearly understood by everyone who has a say in how it should be addressed.
As mentioned, the description in Kaner’s book is much more detailed. The book overall is a great learning tool about testing.
- Kaner Cem, Falk Jack, Nguyen Hung Quoc, ‘Testing Computer Software’, Second Edition, 1993, Chapter 1, ‘An Example Test Series’.
- There are two books written by developers which do a good job of explaining their thinking especially regarding what could go wrong — Everyday scripting with Ruby by Brian Marick and Extreme Programming Adventures in C# by Ron Jeffries. The Ruby community has some good books on the various design tradeoffs when designing code, e.g., books by Russ Olsen or Sandi Metz.