Testing a device before shipping
When we released the Ubi, it was the first time UCIC had released any hardware product. The Ubi wasn’t just any hardware product. It was an integration of many different technologies that were only starting to come of age — speech triggering, speech to text, audio conditioning, text to speech, persistent IoT measurement reading, and a cloud service to manage the experience. Our original “modest” scope of the Kickstarter project had turned into a a colossal undertaking that had stumped even technology giants.
One of the things we quickly learned is that in a device with so many parts, it’s better to test it as a complete system and then to drill down to componential improvements. This is how we set out to test the Ubi as the first devices rolled off the factory line.
Over four years ago, we brought together a group of Toronto tech enthusiasts who had ordered the Ubi and gathered them at the offices of our custom manufacturer partner and distributed the initial Ubis. Of course, their initial use of the Ubi revealed lots of problems and challenges that we needed to address related to setup. We ended up needing to do quite a bit of tuning.
The next step was to ship out the Ubi to the first 50 early backers and then to another 50 after that. This revealed even more issues — namely that there were tons of different WiFi routers out there and some had issues with certain Android devices. What if the WiFi router didn’t have a password? Oops! Back to the dev team for more updates…
By the time we were ready for our larger wave of shipping, we had developed our own “sanity test”. This was an 8 hour burn in test where we barraged the Ubis with trigger and query over and over again. If you were located close by the test setup, it was torture! “OK Ubi… weather” 200+ times an hour. Plus incessant beeping.
However, that test helped flush out any problem issues that could be encountered within the first few hours of setup. Those devices that would fail would fail quickly.
From what we encountered, here’s a checklist for a first time hardware maker project:
- Test the first devices internally in your team as much as possible. Flush out issues.
- Test out on a small group of your backers.
- Come up with automated tests to flush out bugs in the devices.
- Ship in waves. Pause, observe, repeat.