Enterprise Software Testing: Do you have what it takes?
🖥 🖥
Like most juniors in college, I donned my only tie and bargain bin suit to attend a career fair with a stack of resumes in hand. As one would expect of any computer science major, I visited the booths of many tech firms looking for an internship where I could write some code, acquire some skills, and ultimately secure a full time offer. All it took was a single conversation with an IBM Distinguished Engineer in Test to change the entire trajectory of my career. He asked a seemingly simple question, “What is the hardest part about a project you have worked on?”. Rather than diving into null pointer debugging or optimization issues, I responded with a more generic theme, “…to make sure they always work and no matter how my professor uses it or what they enter as input…which has sometimes made me completely start my project from scratch…”. With a small grin he said “Did you ever think about a career in testing?”.
How would you test a pencil sharpener?
Similar to the “sell me this pen” part of a sales interview, one common technique for a test or quality assurance role is to ask “How would you test a <some object>?”. In the case of say a pencil sharpener, the first response most have is to see if it works or sharpens a pencil. Some respond with a few more of the simpler test cases that begin to quantify performance like “how sharp does it get?” or “how fast does it sharpen?”. However, the following couple of responses are often the most important and telling in revealing a candidates traits. If a candidate begins to ask questions about the details of the object to understand context, it can indicate that they are curious and imagining various possibilities to further refine specifics. For instance, it matters if the pencil sharpener is mechanical or electrical, as well as if it can sharpen different sizes of pencils. Having this context, one should then be able to articulate a few more tests to be considered like testing various sizes, shapes, or types of pencils. Context is key to understand any object under test, what needs to be focused on, and the potential weaknesses. Gathering information and being outspoken enough to ask seemingly silly questions, is a significant part of any test role.
Think Outside the Box
Whether architecting a comprehensive test plan, defining testcase variants, or coding test automation, a tester must attempt to ponder and consider all eventualities. To do this, testers are creative, detail oriented, and think outside the box. The question “What if______?” should continuously be asked about designs, during code reviews, while developing testcases, and especially during execution. One can not just consider the intended and documented use cases, also known as the “happy path”. A tester needs to think about how something could be used or exploited in potentially unintended ways. A testers mindset should challenge assumptions and think a bit unconventionally. Going back to the pencil sharpener example, what if someone puts the pencil in eraser first? What if someone puts in a pen? What if the shaving catch is full or not installed? These types of “what if” scenarios map to situations where users invert usage, skip steps, or run out of resource(s).
Lighting the Box on Fire
Testers should embrace their mischievous and devious side to go beyond thinking outside the box and remove almost all restriction and preconceived notions implied by the proverbial box. Ponder that anything can happen, at any time, for any reason. During the first week of my test career, my mentors took me to our portion of the data center to pull out network cables to show the system could recover gracefully and further shutting off the power to show the system could easily recover to an operational state. Situations like these error and recovery examples need to be considered to account for various application, server, and/or infrastructure failures. Extending that to our sharpener example, what if the pencil sharpener is unplugged or plugged in while running? What if someone uses the sharpener upside down? Another example of purposeful deviousness, is through load/stress testing where the intent is to push the item under test to its limits and beyond. Understanding limits is quite a useful exercise to define best practices, identify bottlenecks, and verify documentation. In the case of the pencil sharpener, one can consider how many pencils one can sharpen before the device no longer sharpens or before the motor becomes overheated.
Crank it up to 11
Thoroughly testing any non-trivial product often requires significant skill and resources but mission-critical and enterprise grade testing takes it to a whole other level. Mission-critical grade solutions have availability and reliability requirements often guaranteed by service level agreements exceeding 99.99%. Failures of such systems can impact a large number of users, introduce a potential market shift from eroded reputation, result in significant revenue loss (thousands to millions of dollars lost), and may even introduce legal challenges. Creating systems and solutions that perform at this level requires diligent application of quality considerations and safeguards throughout the entire development process. This includes a rigorous testing strategy and regime executed by highly skilled test experts. Driving this thorough strategy across multiple test environments helps remove latent defects and validate requirements before widespread availability and even continuously afterward. At this level, testers need to employ system level thinking to understand the holistic system and the infrastructure in which solutions operate. Enterprise level testers leverage mathematical models through techniques like state machines and combinatorial test design (CTD) to provide coverage with optimization. Real world workload and activity simulators are intelligently developed to mimic clients use cases, applications, and users. These tools drive load/stress in addition to scalability testing. However, expert testing doesn’t just require a sledge hammer, more often than not it requires tactical precision. Strategically walking boundary conditions, injecting errors, and driving timing windows to expose race conditions and drive recovery actions. For enterprise systems, recovery is critical with redundancy in the hardware, software, and applications. Even going as far as injecting errors while handling other errors is necessary to drive multiple layers of recovery actions. Furthermore, rarely are solutions run in isolation, enterprise testing requires testing of the complete stack and driving iterations across multiple solutions. This cross solution level testing requires expertly architected application workloads crafted for exploitation as if they were being deployed for production. Mission-critical grade testing requires rigorous software development practices of the test as well. A mistake or defect in the test automation or test code can lead to a test escape. Therefore, test should also followed software development best practices like static code analysis and code reviews.
One such example is the IBM Z, a highly scalable, secure, and resilient platform for running mission-critical applications offering. This platform, often called the mainframe, offers up to 99.99999% availability. Yes, that is 7 nines, which equates to about 3 seconds of unplanned downtime annually. To offer such high qualities of service, testing is performed often at extreme but necessary levels. In addition to employing and continuously inventing advanced enterprise software test techniques, extensive hardware and platform level testing is performed. For instance, IBM test teams use specialized chambers to ensure components can operate in unplanned data center conditions like extreme temperature and high humidity. IBM Z uses an earthquake simulation machine to help reduce the possibility that even a natural disaster interferes with mission critical workload. Check out the testing section of a video from a previous generation Z machine.
https://youtu.be/WhIp9kiqSD8?t=617
TLDR;
Enterprise level testing requires dedicated experts with specialized skills. Its more than smashing things with a “big hammer”. These test experts need to be creative, devious, employ system level thinking, and have an almost maniacal level of focus. Testing at the enterprise level requires a deep technical understanding and a surgical precision utilizing mathematical and logic modeling to design tests and the supporting test infrastructure to discover defects efficiently.
Do you have what it takes to become an expert enterprise software tester?
Further Reading
- IBM Z Platform — https://www.ibm.com/it-infrastructure/z
- Software Testing — https://www.ibm.com/topics/software-testing
Disclaimer
While I have the pleasure of being an IBMer in Z Systems, these views are my own.
