Software Testing Life Cycle [Software Development Engineer in Test Article Series Part 7]
This is the seventh article of my “How to Become a Software Development Engineer in Test” article series. All of these articles have been written in ‘question-answer’ format. As I learned from some of my Filipino friends, this method was developed by Socrates to explain things better. So I hope the techy and non-techy guys will benefit from these articles.
You can find the previous articles here. It is highly recommended to read the previous articles before this one.
Since I have been a software developer for more than 11 years I will provide some sample codes of real and working examples. These code samples originated from mobile applications, desktop applications, web applications, web services and various database procedures as well.
To explain things much better, LadyQ and SorBi are intensively used in my articles as samples.
Lady Q: A businesswoman and client for the SorBi app. She pays money to a software development company to have the SorBi app developed.
SorBi: An android application for creating and sharing polls. SorBi is used as an example in this article series. This app is available on Google Play and it supports English and Turkish languages.
Let’s enjoy learning the exciting stuff of Software Testing Life Cycle!
Question: I want to be a Software Development Engineer in Test (SDET). You talked about test types in our previous session. But, I don’t know how to ensure that the software quality objectives are met. How to perform those tests?
Answer: Ensuring the software quality is a collaborative task including all team members. First of all, understanding the client’s requirements is the very first and the most important process. The Product Owner (PO) is responsible for this. Then, converting the requirements into user stories that match the INVEST criteria is again Product Owner’s duty. Designing a good, robust, scalable system is the Software Architect’s responsibility and so on…
Q: So, Why is it said that “software testing ensures the quality of the product”?
A: Because, it is so. Especially static testing starts from the beginning of SDLC as we emphasized our previous sessions. The key thing here is that: the Testing phase is the 4th step of SDLC but it is strictly dependent on the 1st step, that is Requirement Gathering. Because user stories are created from those requirements. And tests are done to ensure the user stories are completed.
But, there must be something more to check whether user stories are completed as expected. For that reason, The PO creates Acceptance Criteria (AC) related to each user story. The software developer, assigned to a user story, codes so as to meet that criteria. The software tester, assigned to the same user story, creates scenarios and performs tests, again, based on the same Acceptance Criteria. So, as you see, this AC is a reference for both developers and testers. AC is also a reference for Definition of Done which will have a great importance for accepting or rejecting the sprint at the end of each sprint.
Q: Well, as I see there exists a lot of steps for testing purposes. Can we identify that specific process for software testing like the process we have in software development (SDLC) in general?
A: Sure, the sequence of specific actions performed during the testing process is called Software Testing Life Cycle (STLC). STLC ensures that the testing process is executed in a systematic and planned manner.
We can perform STLC in both Waterfall and Agile Methodologies. But it is very obvious that, in Agile, STLC is not just a separate step, but a continuous cycle throughout the whole SDLC process.
Q: If it is a cycle like SDLC what are the steps of Software Testing Life Cycle (STLC)?
A: As it is emphasized, the first step of SDLC, the Requirement Gathering, is the most important step. Similarly, we can say the Requirement Analysis (1/6 steps of STLC) is the first and the most important step of STLC.
Q: I guess, I can figure out easily what the client needs. Is it a big deal to ask everything to the client, I mean just do the right thing, ha?
A: That is the most dangerous thought a developer or tester can have. As it comes to understanding the client’s needs you have to understand his/her expectations fully. What you expect is not important. It can be totally different from the client’s expectations.
Since we develop the app based on what the client wants, it is crucial to understand his/her requirements. So, simply, read the user stories and their Acceptance Criteria, make sure you understand them fully in the Requirement Analysis step.
There are entry and exit criteria for all levels in the STLC. In the Requirement Analysis step, the entry criteria is Software Requirement Specification Document. The QA team analyzes that document as well as Acceptance Criteria.
Requirement Traceability Matrix (RTM) is one of the exit criteria of this phase. RTM is a document which shows the match between the requirements/user stories and the test cases. So, from the requirement point of view, you can easily find the matching test cases, or vice versa.
Q: Okay, okay. Let’s assume we identify the customer’s expectations, what’s next?
A: Next, we have to Create a Test Plan (2/6 steps of STLC). Test Plan is a document, mostly prepared by QA Lead. It includes test schedules, testing tools, resources, areas to be tested and not tested, assigned testers, testing types to be performed and so on. The exit criteria of this phase is the Test Plan Document.
Q: Sounds a lot of paperwork. When is this Test Plan Document created?
A: Yes it seems it requires a lot of paperwork. In Waterfall Methodology, since every step is strictly bound to the previous one, the Test Plan Document is created for the whole software, right after the coding phase. But, in the Agile Methodology, since it is an iterative approach to software development, a Test Plan Document is created for each release, regression or sprint. Moreover, sometimes, the test plan is discussed among team members but not documented.
Q: Why is it so important to write everything down?
A: Actually, that is not everything. Moreover, most software companies have templates for each type of test plan. It is not a big deal. What is important is, documentation is mostly about defining boundaries. In the 3rd step of STLC we Create Test Cases (3/6 steps of STLC). This is also a documentation. They can be written in an Excel file, or saved in an issue tracking software like Jira.
Test Case Creation is performed based on the Acceptance Criteria of the user stories. Acceptance Criteria is again mostly related to defining boundaries and behaviors. Before creating a Test Case (TC) a tester has to think about the possible scenarios based on the AC. Let’s examine a sample flow from user story to scenario:
As seen in the image, the tester thought about the first Acceptance Criteria and created both negative and positive scenarios. These scenarios will be transformed to test cases. Every test case has some information.
Let’s create a test case based on our sample scenarios:
Test Case ID: Automatically given by bug tracking softwares like Jira.
Test Case Description: Verifying that the user can log in with a valid username and password.
Precondition: Testing environment should be ready (e.g. QA Environment or Staging Environment). Test data should be ready (A valid username and password should be provided by developers or database admins).
Steps, Test Data and Expected Results are shown below:
Actual Result: The result we have after performing each step.
Pass/Fail: Test condition based on the actual result and expected result match.
Comments: Optional area for testers to write comments.
Screenshots: Evidence for each step to show what testers have after performing each step. Like screenshots, database query results, screen recordings, log files, etc. can be saved in the test cycle closure documents to support the completion of test execution.
Bug Ticket Number: If there is a bug, a bug ticket is opened by the testers. This bug is related to the current test case.
It is critical to link the test cases with their user stories. They must be reachable from user stories, and user stories must be reachable from test cases.
The exit criteria of the Test Case Creation phase is test cases, test data, and, if applicable, automation scripts.
Q: It makes sense. Where to perform those test cases?
A: As we know from the SDLC, there are four environments in software development; (1) Development Env. (2) QA/Test Env. (3) Pre-production/Staging Env. (4) Production Env. Those environments are combinations of hardware and software.
The 4th step of STLC is Environment Setup (4/6). Each environment has different test types.
- Unit and Integration Tests are performed in the Dev. Env.
- Manual Tests, Functional Tests, Smoke Tests and Backend Tests are performed in the QA Env.
- Regression Tests and UAT Tests are performed in the Pre-production (Staging) Environment.
The environment in which a tester does tests must be ready; operating systems and virtual machines are configured, testing tools like JUnit and TestNG, and databases are deployed. The test team performs a smoke testing of the test environment.
Q: Let’s do the tests. We waited so long, didn’t we?
A: Oh, yea. It is a long process until executing tests. Test Execution (5/6 steps of STLC) is performed by both manual testers and automated testers. After executing tests a tester has results from them. Those are called Actual Results. Remember, The Expected Results are already written in the Test Case. Now, it is important to compare actual results and expected results. If they match, the tester can define Test Status as PASS, otherwise the tester must perform extra actions.
Q: If the test can not pass, then it means it has failed. What’s more?
A: That is not always true. The test can fail because of bad coding or code inconsistency. Moreover it can fail because of bad data, the tester used while testing, or, if it is an automated test, the tester’s code may cause the error. So it is obvious that the tester must do his/her duty with all aspects, before defining a bug. The tester checks his/her code and process. Then, the same error must be reproduced at least three times with different data. Only after that tester can define the Test Status as FAIL, and he/she can create a BUG TICKET. That Bug Ticket must be linked with the related Test Case as well as related User Story. When a user story is selected from a issue tracking product, one must see the related test cases and related bug tickets. That is very important for tracking the issues.
Q: How can we define a bug?
A: Any mismatched functionality found in an app is called a Bug or Defect. That’s why we have Acceptance Criteria. It is a reference for both developers and testers as emphasized earlier.
Q: How to create a bug ticket and why, just directly inform the developer?
A: Creating a bug ticket or reporting a bug is a very important process for STLC. In fact, there is a process called the Bug Life Cycle.
Q: Oh my… Cycles, cycles, cycles…
A: No need to panic. It is a straightforward process. First, I must talk about bug reporting.
When a defect is detected, it is the tester’s detection. But, in some cases, the developers disagree with testers. They can say that is not a defect etc. So the last word belongs to Product Owner. That’s why a bug ticket is created.
Moreover, sometimes, the developers have no time to fix the bug and bug fixing can defer to the next sprint. So there must be a bug ticket to follow and ensure it is fixed.
It is also a good documentation for the software development process. In case there are new developers or testers in the project, they can look at the past actions in a tool like Jira, and understand the process.
Once a defect is logged, it should be mapped with the related test case and corresponding requirements in the Requirement Traceability Matrix. Therefore, these documents are updated in that phase. Updated Test Cases, RTM and bug reports are exit criteria of the Test Execution phase.
Q: Now, it makes sense. It is important to create bug tickets. Please talk about the Bug Life Cycle in short. I do not want to learn any cycle from now on.
A: Bug Life Cycle is a systematic process for dealing with defects. The flow of bugs in this cycle is represented by bug status. These are possible bug statuses in the different stages of the Bug Life Cycle:
Scenario 1 ends with closing the bug ticket or reassigning the developer based on the re-test action’s result.
In Scenario 2, the assigned developer rejects the bug ticket because of a duplicate bug issue or not admitting that it is a valid bug. Then, the bug status changed to closed.
In Scenario 3, the fixing of the bug is deferred to the next sprint, so the bug status is deferred.
At the end of the Test Execution phase the test cases are updated with execution results. Defects should be closed or deferred.
Q: Very well. Are we done with the STLC?
A: One more step: Test Cycle Closure (6/6). At the end of the STLC test reports are generated. These reports should include the time spent, the percentage of errors found to positive test results, the total number of errors found and fixed. Test metrics are prepared based on those parameters. Bug statuses are defined. If the testing department is concerned, test cases, test results and metrics are evaluated for improving the productivity of future tests. Test reports, test metrics are the exit criteria of that phase.
Q: Is it possible to have a %100 bug-free software?
A: A bug is unknown until it is detected. You can not know whether there is an unknown defect. But, believe me, there is always at least one defect, even if you think you’ve found the last bug. And the fix of that bug might give birth to new unknown defects. Think about that question again.
Q: 🥺
A: Do not worry. We are the bug patrols. 💪👾💪
Bugs are welcomed as long as we can find them. They are the main reason for us to find a job. Get along well with them. 😀
