How Is A Product Tested In The Real World?
by Akshay Anand @ www.dotsandarcs.com
Most of the software testing in the world is outsourced. I live in India and I presume that a majority of the most popular web products are tested in India. I am among the few lucky ones who recently received the opportunity to test a real world product from its inception to its launch.
During my graduate studies, I was taught about multiple buckets of testing. There were so many concepts — white-box and black-box testing, integration testing, sanity testing, core testing, component testing, and the list goes on and on. And we used to mug up these definitions for our job interviews. But here’s an eye opener for you if you’re a graduate and reading this — the real world is very different from what you read in books.
So let me narrate my story, and tell you the best practices for testing a real world commercial grade product. Right up-front, “A stitch in time saves nine” applies perfectly to testing products. Testing should start as early as possible (from Day 1, in fact, if possible) if you plan to deliver a high quality product. A good testing method should be agile, which is to say, the testing should go hand-in-hand with product development. Our team started testing on Day 1 itself.
But hold on. What did we even test on Day 1? There was nothing available, there was no functionality built, so what was there to test?
Well, for us, we first went through the wireframes and mockups, and gave our feedback on the user-interface. Feedback is imperative, and you will see me talking about this multiple times in this article: as a tester, our most important insight is to give feedback as a real user of the application. Should a color of a button be altered to make a feature more visually appealing, or should a work-flow be changed to make the product more intuitively usable? These are some examples of early-stage testing.
Once development is underway, it is time to test the functionality of the product. This phase is equally important from the perspective of a test engineer, as there might be multiple scenarios related to each function. The developer shall only test the most basic scenario, and its upto the test engineer to make sure that the functionality works in every scenario.
For instance, to test the functionality of a simple button means understanding and testing it under various states of the application that are possible before the button is actually clicked. Of course, then you must check if the button performs its intended function. You must also test the web console and server logs to ensure that there are no consequent errors. Then, there might be specific scenarios and use-cases like testing while you are logged-in and logged-out, while you are on a paid plan vs. on a non-paid plan, etc. All in all, testing a simple button can easily become half a day’s job. The task becomes even more complex when we have to test the application on every different machine that it’s supposed to work on.
“I don’t care if it works on your machine! We are not shipping your machine!” ~Vidiu Platon
I’m sure you’ll relate to that. ;)
Next comes performance testing. With respect to our application, as it became more and more stable, we also started testing its performance. I remember instances where all ten of us in the office would sit and simultaneously upload a file on the application just to see if our server could handle the load, and if it could, how much time did it take for all the files to finish uploading. Over time, we created performance matrices and started updating them multiple times a week.
If you have done all of the above, then by this time, your product is probably in really good shape. The final stage is to focus all your energy on testing the system in its entirety. This means that you include the additional variability of the server where the web product is intended to be deployed. Also, remember to take into account the web browsers and devices on which the user might use it. Basically think of the complete end-to-end scenario, including both the hardware and software used, and test the product in its complete life-cycle.
Finally, the last step is automation. I know that there is a lot of excitement and noise about automation and every engineer and company wants to pounce on it. But based on experience, I’ll tell you that automation should be employed very late in the testing cycle, only when you know that your application is stable and will not change much. Writing test cases takes a long time, and updating them can take even longer. So trust me when I tell you this — automation tests should only be written when you’re sure you won’t need to change them drastically!
So this is what I did. I tested a real application through its entire life-cycle at dots&arcs. Now the application is live, and has live users. And not to beat my own drum or anything, but I am pretty proud of what we did with the app. Witnessing the conception of an idea and helping it grow into something wonderful is an experience of its own kind.
“Quality is never an accident. It’s always the result of intelligent effort.”
I have learnt that while testing may not be rocket science, it sure isn’t something that can be learnt through books either, at least not the real deal. Testing is just common sense. A good test engineer thinks of ways in which the real user would use the application, and then performs the same activities on the application. Testing is about sharp observation a lot of patience. So while I’d really like to tell you that I have some secret ingredients of testing with me, I really don’t!
I’m like Panda Po’s adoptive dad, Mr. Ping, who does not use any secret ingredients in his magically tasty noodle soup, but simply believes its special. ;)
by Akshay Anand @ www.dotsandarcs.com
