Software testing is the process of trying to discover every conceivable fault or weakness in a work product. The purpose of testing is to find errors.
Software under test is a part or subset of the program we have completed, where we can exercise some behaviour. This is some module or unit of code (something small, like a method, a function, a subroutine, a procedure).
Test data (the input) is the information to run a test against software, it is just the input to the program. You can generate test data based on a profile of how you think the user will act based on probability or user studies. Or you can attack the code with some data that often causes errors (inputting zero, a value that is larger than what it asked for, a word instead of a number). For each of these test inputs, you run the software under test. Before running the test, you set up the software to be in the right state, so test data makes sense when it’s inputted.
Output the behaviour of the program given that input.
Oracle is the developer or the tester, who’s running the tests. The tester inputs the data, runs it, watches what happens and decides whether the behaviour matches expectations. However, humans aren’t reliable. So, we’re starting to see automated oracles, which compare some known, or determined, or retrieved expected output to the actual software-generated output. For automated oracles, you need input and output (test cases), that way a computer can read all those pairs and run the test quickly.
Test cases are the test data (inputs) to the system and the expected output from operating the system on these inputs, if the system performs to its specification.
Failure occurs when the delivered service deviates from the specified service (something didn’t happen the way it was supposed to). The failure occurs because the system is erroneous, the failure is the manifestation of an error.
Fault is what happened.
Error is the part of the system which is liable for leading to a failure. It’s whatever is wrong that leads to delivery of an output that doesn’t comply with how it’s specified to work. The error is a manifestation of a fault.
Latent error is the error that sits in the code and hasn’t been activated yet. Upon occurrence, a fault, a mistake create a latent error. You can have latent errors for years before an input runs the code in that way and it produces unexpected behaviour.
Effective error is the error, which affects the delivered service. The effective error causes a change in behaviour that the user did not expect to see (a failure).
Example 1. Programmer makes a mistake (it’s a fault). The consequence of that fault is an erroneous instruction or a piece of data that is defined improperly (it’s a latent error). Then an appropriate input pattern activates the module with the error (it’s a latent error becomes effective). Then the effective error produces erroneous data or behaviour or something happens not the way the user thinks it should happen (it’s a failure).
Example 2. A maintenance writer makes a mistake in the manual (it’s a fault). The consequence of that fault is an error in the manual directives on how to use the software (it’s a latent error). This error remains latent as long as the directives aren’t acted on. Then someone reads the manual and tries to run the code using the erroneous manual, but because of the error in the manual, the code doesn’t work the way this person thinks it should (it’s a latent error becomes effective and leads to failure).
Verification means that software conforms and performs to its specification, you are building the thing right. There’s a big transition between what the user wants and what you said you’re going to build. You write down what you want to build, give it to the team, build the system, and then verify that what you built matches what you wanted to build. Verification is a lot easier and cheaper, than validation.
Validation means that software conforms to user satisfaction, it’s what they wanted, you are building the right thing. However, what users say they need and what they actually need can be different things. You have to validate, that a) this is what they need (watching them using the solution, making sure it works the way that you think it will), and b) they are satisfied with the way it provides what they asked for. Questions to ask yourself: Are we building the right thing? Does the product do good things? Is it good for usability and security? Is it good in a user satisfaction level? We do this through dynamic and static validation and verification (V&V). Validation is more difficult and more expensive than verification.
Dynamic V&V is exercising and observing product behaviour. It’s running the system and seeing what happens. That’s all forms of testing.
Static V&V is about looking at established representations of the solution that aren’t necessarily running the system. It is looking at the code and seeing things right without running it. For example, a code inspection, a code review, or pair programming. It’s also formal proofs, model checking, mathematical type of proof for solutions of code. You see a lot of that in formal verification, formal testing, in terms of security-based testing (penetration testing, trying to get in the system when it’s running).
You care about building the thing right and building the right thing. In order to do that, you have to employ a lot of different testing methods.
Verification and Validation Process (V&V Process) is a whole life cycle process, which should be done at every step. This is more true in agile teams, where a user representative is a key component of a team. At every step of the project, users should validate the work after the developer has verified that they have done what they planned.
V&V Process has two objectives: 1) discovery of defects in a system, 2) assessment of whether or not the system is usable (Examples: How well does the system work when the CPU is overloaded? If there are a lot of users at the same time, does the database connection usage reach its limit? What happens if a server shuts down unexpectedly? How easy is it to hack into the system and corrupt data?)
Black-box testing is designed without knowledge of the program’s internal structure, it is based on functional requirements. You should know nothing about the code when determining how you will test the system to ensure it’s working properly. Follow behavioural approaches. If I do this, will it do the expected way? How quickly will the system respond? How resilient the system is?
White-box testing examines the internal design of the program and requires detailed knowledge of its structure. It is about making sure that you test all the lines of code and attack fragile or error-prone elements that had been used in the program.
Stages of testing
Unit testing is testing of individual components (an individual method or one class within the solution).
Module testing is testing of a set of units that come together as a collection or dependent components. This is the first form of integration testing when things come together.
Sub-system and component testing is testing of collections of modules integrated into sub-systems. You are ensuring that the interfaces between the sub-systems and components, between the module’s components (often divided between developing teams) meet the specifications and work together as intended.
Full system testing is testing the complete system prior to delivery. Making sure that the whole system is working, testing security, performance, usability etc.
Acceptance testing is testing by users (alpha testing, beta testing), and testing after deployment when the users find bugs and report them to you (it’s the worst testing because you don’t want the bugs to get to real users).
Software testing strategies
Incremental testing
Incremental testing helps to determine if something has changed in the previously correct code based on something you added. It also shows whether the current modules still work as intended.
In this testing, you test each module individually in the unit testing phase, and then modules are integrated incrementally and tested to ensure smooth interface and interaction between modules.
In this approach, every module is combined incrementally, one by one till all modules or components are added logically to make the required application, instead of integrating the whole system at once and then performing testing on the end product. Integrated modules are tested as a group to ensure successful integration and data flow between modules.
Regression testing
Regression testing is the technique of re-running older tests in a larger suite. That’s a big part of incremental testing.
Top-down testing
Top-down testing is the technique when you have to develop something to stand in for the elements at lower levels that you haven’t created yet.
They haven’t been built yet, but you still need to be able to do those tasks to make sure that the program works. One of the things that you can do is write a stub. A stub is a few lines of code that just returns a hard-coded value that stands in for a real return value. Stubs are created to allow higher-level units to execute.
The same kind of thing can be done with a mock. A mock is something where you don’t hard code something, you just check, if this method was called. If yes, you move on.
As you move down you continue to build levels of software down and stubs below those to continue your work.
Pure top-down doesn’t exist.
Bottom-up testing
Bottom-up testing is the opposite case of top-down testing. Pure bottom-up doesn’t exist.
Bottom-up testing is the technique when the lower-level implementation is complete, but you don’t have the larger picture integration execution drivers, therefore you need to develop something to stand in for the elements at higher levels that you haven’t created yet.
These drivers walk through the process of what possible calls to lower-level elements might be and make reasonable calls to ensure that lower level is operational. Drivers organize and execute the units under test.
You make your best assumption of what the most common, or most important orders of operations are going to be and make sure that all your lower-level operations are complete.
Back to back testing
The idea is that the program worked before. For all the things that worked before, you run test data for that working behaviours through both the old version and the new version. You do a direct comparison of the output. The outputs should be the same since it worked before and it should continue to work. Alternatively, anything that developers have modified to fix something or add some feature, you run the test data through both iterations again to make sure that they are different. It takes some manual inspection to see that the change in the result is what you actually wanted to change and that it changed in the right way. But at least it’s a start, especially when you’re working from scratch you don’t have any automated tests from the beginning.
Back to back testing helps to use earlier iterations of a program as an effective automated Oracle. It expands test data without including expected output.
Test scaffolding
The goal is to set up an environment for executing your tests. The environment consists of:
- Driver — initializes non-local variables, parameters and activates units under test.
- Stubs — templates of not working modules used by the unit.
- Program unit.
- The Oracle — the human or automated oracle, which verifies the correspondence between produced and expected results. It verifies that stub, driver, and program unit operate properly.
Effort in test execution vs. Effort in developing drivers and stubs
Who should test
Assign your best developer to test the system, to be a tester.
A developer understands the system, a tester must learn the system.
A developer will test gently. A tester will attempt to break the system.
A developer is driven by deadlines. A tester is driven by ‘quality’.
Axioms of testing
- As the number of detected defects in a piece of software increases, the probability of the existence of more undetected defects also increases.
- Assign your best programmers to testing.
- Exhaustive testing is impossible.
- You cannot test a program completely.
- Even if you do find the last bug, you’ll never know it.
- It takes more time than you have to test less that you’d like.
- You will run out of time before you run out of test cases.
Software testing is a process to find defects in your implementation and, hopefully, it will lead to their fix. You can never show the absence of bugs through testing. No amount of testing will ever be sufficient to prove that software works. The purpose of testing is to prove that a program is no good, we can never test to prove that it’s good.
You’ve got to make sure that you have quality inputs, and you have the correct expected outputs.
Each software testing perspective has its own benefits. Use them all, but even all of them combined will likely not be sufficient.
When there is a question “Should we release this software?”, testing is the stage in the process where you say “No, we can’t release yet because these things are still wrong.” So, it is a very important part of the process and also usually very time-constrained.
