Regression Testing in Mobile Development
“Didn’t we fix this bug before?”
Table of Content
1. Introduction
2. What is a regression test?
2.1. Regression set
2.2. Regression Packet / Release Candidate
3. Where did the need for regression testing arise from?
4. What are the benefits of regression testing?
5. How to do a regression test?
5.1. Set up before run
5.2. Create a regression set
5.3. Execute regression test
5.4. Finalize
6. Conclusion
1. Introduction
It is perfectly understandable to break the whole app while writing code in the development phase, in your own branch. But after releasing, a new feature should never break the app’s integrity or a new bug-fix should never introduce some more new bugs.
While we are developing our code, we may think that “My code does not have any conflict with other parts of the app or I’m only working on this particular part of the app”. However, without knowing, we might have caused a bug, while changing a simple property. Especially if the code is spaghetti.
Some typical bugs might arise just after newly added features or bug-fixes. We will discuss these in future articles.
Development teams have come up with a sturdy solution to prevent waste of their hard work and consequently, the failure of their app. And that is simply, “Test the app from the start till the end, every single functionality.”.
Before publishing the app, it has to work as expected. Doesn’t matter what changes inside.
2. What is a regression test?
Regression test is a type of software testing to check whether the post-development has produced a new error, after adding a new feature or fixing a previous bug.
There is no big difference between regression testing of a mobile app and regression testing of a web app. Because, regardless of the platform, the majority of functionalities of the app will not vary.
For example: It is expected to work in the same way on the mobile platform as social login (login with facebook) works on the web platform. This means that functional aspects of the application do not make any difference in mobile regression compared to web regression tests. In fact, some mobile app regression tests can be used in web regression tests and vice versa.
Mobile regression testing has its own challenges. Below are a good list of these:
- What is the screen resolution?
- Is GPS on or off?
- How consistent is the application in portrait and landscape view?
- Which brand and model of device?
- How does the application perform in mobile interruptions?
- Is the mobile application properly getting installed on every version of mobile devices?
- Is the usability aspect of the mobile application up to user satisfaction?
- Is the layout and navigation of the application still user-friendly?
This list can grow bigger based on the OS, country, currency involved, etc.
2.1. Regression set
Regression set is a bunch of test cases that the software requires to pass to become a candidate to release. Test cases should cover the whole crucial aspects of the app.
A potential regression packet must pass the whole regression set. Otherwise, it has to be fixed before proceeding.
Regression set, is a living document. Always updated according to the features. Regression set content may differ in software.
2.2. Regression Packet / Release Candidate
Once the app passes the regression set, now it can be verified as a Regression Packet or Release Candidate. We have reached the checkpoint. It means we can publish this version.
We can use these examples as naming conventions. We know that these are reliable builds.
Example: Android 3.3.2-rc, iOS 2.2.1-rc
3. Where did the need for regression testing arise from?
Let’s say we have an application that works perfectly. Tests have been done. The probability of users encountering a bug has been reduced to the lowest possible level. Everything is ready and we published it. No complaints from users. Life continues well.
Let’s say a feature is requested, and that feature has been developed. Functional tests were carried out for that feature. The development merged to the Master branch. And we see our new feature is still working.
Now. We had an application that ran before. Besides, we had a new feature that has passed functional tests. The combination of these two, must be an application that works properly. Innit? Unfortunately, it is not always the case.
The integrity of our development may have been disrupted in a way we do not know. Hypothetically, the risk of encountering bugs has increased. And we don’t know that, because we haven’t checked it yet. A freshly merged feature should be seen as a potential source of bugs. An app that has been thoroughly tested turns into an untested app after a feature is added to it.
Releasing a risky app to the market is not the job of successful technology teams. The way to find out if the new development has created a bug compared to the complete app is through regression testing.
4. What are the benefits of regression testing?
A passed regression test means that we now have a checkpoint for the app. We are making that version of the app “safe to return” or in other words “working version”. In the future after merging features, in case of a failure, we have our safety net.
We need to see our app with it’s totality to plan future steps. A version with passed regression tests provides that for us. Our team would not want to decide on future features based on a risky (potentially buggy) app.
5. How to do a regression test?
5.1. Set up before run
We should answer below questions before proceeding:
- Which devices do we have?
- How many types of OS do we support?
- Are the test environment and test data suitable for regression testing?
After we answer the above questions, we should know that our resources are sufficient or not. For instance, if we have iPhone 7 and iPhone X for testing physically, it means that we are going to test iPhone 11, iPhone SE, etc on a simulator. Same situation applies to OS, brands, and environment.
According to our answers, we may decide to postpone the regression test for a more suitable time. Because it doesn’t make much more sense to do a regression test without missing mock data.
5.2. Create a regression set
The test cases included in the regression set is a list in which a tester will determine whether a software meets all the requirements. Test scenarios are of great importance in regression testing. If we can create a regression set with precise inputs and outputs, we can accurately test each piece of the software and catch errors easily. If we test the software without proper test cases, parts of it might be overlooked. Therefore, we must create test cases correctly. While preparing the regression set, we should be able to give the following information.
- Which platform,
- Any special conditions (such as “This test case will not be included in iPhone 4”),
- In the scenarios, the input-output state is specified.
Below are tiny regression set examples for Android and iOS apps. Three dots indicates that the list goes on.
Android Regression Set
…
While Logout;
Home page scenarios
…
Some conditions may be as follows;
This improvement has not been made to Huawei phones.
LG phones should be excluded for vertical / horizontal direction tests.
For phones with screen resolution below 1080p, this scenario should not be tested.
iOS Regression Set
…
While logged in;
Favorites page scenarios
…
Some conditions may be as follows;
This improvement was not made to iOS 14.
If texts do not fit on iPhone 5 devices, they should come as “…” at the end of the line.
5.3. Execute regression test
We perform the regression test in accordance with the following steps in order to proceed with a planned move and prevent time losses.
- After development, first analyze the changes, identify the affected areas.
- Determine the starting point of the regression. This is up to us to pick. With the app experience, we can determine multiple or a single point start. For example:
Not logged in cases, starting from home page.
Logged in cases, starting from My Profile page.
3. Start to run the regression set.
4. Check cases that pass the test.
5. Describe the errors with detailed explanations in case of errors.
6. Wait for the developers to fix bugs.
7. Verify if errors are resolved.
8. Run your regression set again.
9. Determine the stopping point of the regression. The ending points are usually external dependencies. Since we do not own the codes of dependencies, we can test them last. Because we need to find our bugs. For example:
3rd party login cases.
Push notifications.
10. Finish.
5.4. Finalize
Congratulations. Now we ran the regression set and completed it successfully. The release candidate package is ready. After this situation, what needs to be done is to get the approval of the product owners and upload it to the store.
6. Conclusion
- Regression test is a checkpoint for the development. It makes a version safe to return.
- Regression set is a list of test cases that we determine. In order to call the app version reliable, it has to pass each test cases.
- Environment, devices, server, and data are equally important and must met the requirements before starting regression testing.
- Regression testing can become a heavy duty and is proportional to the scale of the project. It’s okay if it’s going to take one to three days.
Hope this article will be useful for you. Have a nice test.
