The Right Way to Test Your New Software Features
I have worked in many jobs and industries over the years and all of them have involved some kind of software or online product for users. I have used and observed many different testing ways and methods over the years. Below are many of the tools that I have used and lessons I have learned over the years about testing, and lack of testing.
Why are you testing?
This is an important question to ask yourself. What do you need to test? What happens if you don’t test? Having the correct understanding and motivation for testing I believe is crucial. Having bugs in your software can be costly to your company. The monetary costs to your company, as well as to your users can be huge. We see this type of thing all of the time in software. Some bugs maybe annoying or inconvenient to their users, where others can be more serious and can cause loss of life.
- August 2012: Knight’s Capital took a $440 million loss after a “glitch” in their trading system. The “glitch” sold all the stocks it accidently bought
- August 2003: A programming error caused a blackout in the Northeast part of the United States. Estimated losses of $7–10 billion.
- 1985-1987: A computer controlled radiation machine called the Therac-25 overdosed six patients, killing three of them.
- April 2014: Hundreds of flights out of LAX were cancelled or delayed due to insufficient computer memory at air traffic control.
What types of testing?
You and your team need to decide what types of testing you will need to, and what you are capable of testing. This could include all of any of the following: alpha testing, unit testing, integration testing, system testing, sanity testing, regression testing, beta testing, stress testing, load testing, etc. (For a great full list, this site has some great information).
I am going to cover the main types of testing that I have personally worked with over the years at several different companies. This is going to include alpha testing, beta testing, and regression testing.
Alpha Testing
Alpha testing is done when your developers have finished building the new feature but before it is ready to be beta testing. In my experience, product managers, project managers or some other internal team should be doing the alpha testing. Dev has done their testing and someone is needed who is not biased to the code to do another round of testing.
Beta Testing
This type of testing is done by end users. So you will need to determine who are the end users of your software. This could include internal and external customers.
Internal vs External Beta Testing
For internal testing (testing with users within the company who either use the software, or support the software), I typically try to get a good sampling of each team who is using the software. One of my biggest goals is to find the different ways that users are using the software. This solves two goals for me with the software: find bugs that were not found during alpha testing and research on how the software is being used. This research is critical when we are deciding our product roadmap for future enhancements.
Another expectation I have of my internal testers is to try to find ways to “break” the software. This can include functional and non-functional testing. An example of this would be, if I am testing a payments system, I would include a number with more than two decimal places. Or I would include an astronomically large number or small number. I would try to include a non-number. To assume a user is going to use your system in the way you intended it is a very false assumption to make. You need to think of all of the potential ways that someone could potentially make inputs.
When I typically run a beta test with external users, I will first vet them as best I can to make sure they understand what beta testing is, and to make sure they will make good beta testers. I have been burned a few times by external clients who were all excited to beta test, only to find out they they were not good candidates. Make sure you come up with a list of requirements that you have for your external users to determine if they will be a good candidate.
I tend to give the users a limited amount of information as possible. I do not want to bias the testers. I want them to use the software in the way they are used to and not in the way I want them too, or assume them to be using it. Below is some of the information I will typically give them:
- What they are testing
- Training on how to use it
- Features I need them to test
- Limitations (if any) of the new feature
- How to reach out if there are issues
Periodic check-ins with your external testers is critical. I will often get on the phone with them and walk them through a few tests to make sure that things are looking and functioning as they expect it too. Having those phone conversations (if possible) with your some of your users can also help you to understand more of how your users are using the software, as well as you can clear up any miscommunication fairly easily and quickly with a phone conversation versus an email exchange.
Regression Testing (Automated and Manual)
Regression testing means testing your system in entirety to make sure that your new code did not break any old code. This type of testing is crucial. If you do not have the teams and tools yet set up to automate some of this, then it must be done manually. This is where your test scripts will be all important.
Test Scripts
Test scripts are one of those things that you should consider to be a living document. Every release you need to use and revise your test scripts. What new features were added where new testing needs to be added? What testing has just been left out? Have you had other key stakeholders review your test scripts to make sure nothing has been omitted? I have found the most helpful way to revise and edit my test scripts is to do a full regression test of the software. I would encourage all of your testers (internal and external) to do this at least once every release cycle.
Another important aspect of your test scripts, is to test them to make sure they are still relevant. I can be important to include load and security test scripts. You should include tests that should potentially make your software fail. This includes both manual and test scripts used in automated testing. I have worked with teams who have done automated regression testing. They went years without updating anything. We did an audit of their test scripts, only to find out that they were in desperate need of updating, and many of them gave false positives (or negatives).
Logging Testing
Along with the test scripts, it is very important to document how the testing went. This is typically done along with the test scripts so that the result of every test can be documented. It is important to document and keep a log of when a test failed, how and when it was resolved, and when a test passed. I can’t tell you how many times I would often think that this documentation was tedious and a waste of time, but so many times I have gone back to that documentation and found information that I needed. Dates, descriptions and resolutions are very important.
Limited Release Testing
At my current position, we will often release new features in batches to our customers. We will send the release out to a select number of customers, and continue that rollout every couple of days, until we roll out to our entire customer base. We use this as a final test in our testing phase.
When To Roll Back a Release
Sometimes testing and releases do not go as planned, when with the best laid plans in place. It is important to always have a plan in place to rollback a release if necessary. These factors need to be decided between you and your team members. Some factors to consider: customer impact, financial impact, severity of the issue, ability/timeline to make the needed fixes, etc.
Conclusion
So many times testing can take a backseat to everything else in the product and development cycles. But the potential risks of improper testing is really just too high. The time and costs put into creating a testing plan for your software is well worth the effort and will always pay off. The time put into creating, using and maintaining a testing plan will always save time versus dealing with and fixing bugs that were not found in testing.
