Quality Assurance is…Awesome?!? (Part 2)
Our tools are laid out and the goal is clear: to make our product function flawlessly in every situation it may find itself in, while being a joy to use for all different types of customers. With these goals in mind let’s dive into the different types of testing, from the deepest depths of back-end function all the way to high level user experience interaction.
Build Acceptance & Feature Testing
During the development of a product, each release of new software needs to be run through a minimum of diagnostics-style tests before even starting to test the new changes. Does it launch? Do basic updates work?
Once it passes that checklist, I get into testing the new features, specifically focused on the area that’s been changed. I work closely with the developers to quickly find and fix any issues that come out of the changes.
3rd Party Product Analysis, Troubleshooting and Collaboration
We use a 3rd party microprocessor from Particle for handling all things WiFi. Unfortunately, any time you rely on someone else’s code and hardware there are risks. The vendor may not have designed the product to fully accommodate your needs or they may have overlooked use cases that could expose weaknesses. This is pretty normal, even when working with well-established companies.
To make sure we get the most out of their product and aren’t surprised by anything, we have to test and ensure that any issues are understood and can either be fixed or worked around.
A big part of this testing is working with the manufacturer to help them find issues that are difficult for them to spot, since we have our own special hardware and code that may use their product differently than they anticipated. We log issues and help find the simplest ways to reproduce the problems so that they can be isolated and fixed on their end. Then we retest to make sure the fix was thorough and complete!
Regression Testing
Before releasing a product to the public we need to be confident that the entire product has been fully tested — from the app, to the underlying infrastructure that makes WiFi communication with the app possible, to the pump, fan, and light functions. This full regression test is where I map out and then walk through the entire process that a customer would experience.
Stress & Reliability Testing
A main priority is for your Grove to work beautifully from the moment you open the box and set it up, to years from now. To accomplish this, we need to simulate weeks, months, and years worth of actions using automated scripts. Here, I have a simple test that simulates countless changes to your pump schedule:
Sometimes we need to make custom code to really push the limits of the Grove and remove as many surprises as possible even in extreme or ill-advised situations. Sometimes that light control test code turns the office into Saturday Night Fever…
Exploratory
This is the creative, more open-ended aspect of testing, and often the only way to find elusive issues since it doesn’t follow any pre-planned script. This sort of testing involves asking a lot of “what would happen if…” types of questions and then poking around in particular areas of the code or product to answer them.
Manufacturing
Before we ship a product, we do a final manufacturing test to ensure every Grove shipped is fully functioning and set up with the correct settings so that each customer’s experience meets our standards. This test will usually be performed on the manufacturing floor as a final check.
User experience
From the first introduction of a new Grove-owner to the product, throughout the lifetime of the product, the user experience is vital for ensuring happy customers. This means everything from that first reaction opening the box, to regular use five years down the road needs to be flawless in an actual home. This also means taking an unbiased look (or as close as I can get) at the overall look and feel of the iPhone app and the knob used for controlling the Grove. Is it intuitive to use? Does the flow make sense to someone with no prior knowledge of the product? How would a younger or older audience react to this?
Even with the best planning and most thorough testing of a product, it’s never the same as having customers experience and use it in real-world situations. Once we get Groves into customers’ hands, the next step in QA involves hearing from you! As concerns inevitably come in, I help identify the problem and work with developers to find a quick solution. In a lot of ways, you help us make the product better with your own QA. So thanks for that! But now I’ve got some new logs to check out…
Originally published at blog.grovelabs.io.
