FURPS+: Supportability tests domain
Supportability / Maintenance / Operations
// For all categories see FURPS+: Table of Contents
Supportability category has also another aliases: Maintenance, Operations, etc. Supportability is not about Customers Care Support, but it is about software operations support.
Supportability covers test cases on:
- installation & deinstallation
- upgrade & downgrade
- rollout & rollback
- configuration deployment
- start, stop, restart, reload, status calls for the services
- logging
— i.e. everything, that Lab and Operations Engineers deal with.
Installation and Deinstallation
Either you deploy your software manually, with automatic deployment system or you ship it in Docker containers — you need to test it. It is important, because even different versions of the same operating system (for example CentOS 6 and CentOS 7 or Windows 7 and Windows 8) have different set of system / default / pre-installed software that may not allow your software to be installed there. Or your automatic deployment system can have troubles with installation (deployment) or uninstallation (undeployment) of your software.
Installation and Deinstallation is about clean from your software operating system.
Upgrade and Downgrade
While Installation and Deinstallation is about clean from your software operating system, Upgrade and Downgrade is about operating system, that already carrying your software.
These operations often intersect with existing users activity and they are about system downtime. Depending on your system different graceful shutdown procedures must be implemented and tested. Graceful shutdown means, that no new connections will be allowed due to started Upgrade or Downgrade procedures and all ongoing connections are awaited to be terminated.
Upgrade and Downgrade is also about existing users data integrity.
Rollout and Rollback
Rollout and Rollback procedures can affect both features to be implemented and tested and the way they will be implemented and tested. This is especially true for infrastructure components.
For the complex, mature real time systems Rollout of software update may last for months: install a few servers, join them in pool, monitor for a while, update all servers in pool, monitor for a while, deploy for all users in current location, update other locations, continue monitoring during all these phases…
Rollout and Rollback are about minimization of possible negative impact. Rollback is your “last chance to get everything back” — so, don’t forget to design and test it.
Configuration deployment
Sometimes you need to change configuration.
Sometimes configuration could be reloaded on-fly, but sometimes only with a service restart. And the latter is about system downtime and that’s why it should be implemented with some kind of a graceful shutdown procedure (same as with Upgrade and Downgrade). But if configuration is reloaded on fly, then it often makes sense to test procedure with some load on the system / existing users data / connections, etc.
If you use automatic deployment system, sometimes this system can deploy configuration incorrectly — so, you need to test this as well.
Start, Stop, Restart, Reload, Status, …
Start, stop, restart — are commands for manual execution of maintenance procedures. They are used in Upgrade and Downgrade, in Configuration deployments.
Reload is about Configuration deployment.
Status helps in diagnostics of correctness of states of software and helps to debug possible issues. Status also could be used by automatic deployment systems to determine the state of maintained software.
Other commands possible: clear-cache, force-stop (bypass graceful procedures), force-restart, reload-cache, help, etc. — make sure, that software helps to maintain itself manually.
Logging
And finally, Logging. Logging helps Software Engineers to debug problems. That’s why logs must be simple and clear written, with correct error levels. Consider error level will be setup in Production — in order not to miss important log messages in DEBUG level.
Grab your logs during testing for different levels, make sure they are clear and help to debug and at the same time they are not surplus.
I hope you enjoyed reading and found this article useful. If so, please like to recommend others!
If you have anything to add, share, argue or to comment in this article — please, do it! Only in collaboration we can build something really amazing in Software Quality Assurance.
