Why Do IoT Devices Die?

Published in

IoT For All

8 min readDec 3, 2017

We called it the blue ring of death. The Ubi would power on and then be lit up with a blue LED ring, then power cycle again after two minutes. It would repeat the process endlessly or until the user would lose patience. At times, blue ring Ubis would hiss or crackle when they reset, or let out a faint high pitch squeal from their speakers.

The Ubi was UCIC’s first product — a piece of consumer electronic hardware that we launched on Kickstarter in 2012. It was a voice activated WiFi computer and worked like the Amazon Echo but was launched two years before the Echo was announced.

As first time device makers creating a new category of product, we were faced with many hurdles to overcome and we not only managed to develop, build, and ship the product, but support and guide first time users through the setup as well. We learned first hand that hardware devices fail in the field and how to quickly investigate the cause of the issues and find resolutions.

IoT devices pose a unique risk to both those who market and sell them and those who use them. With more devices being added to homes, the issue of devices that stop functioning is growing. These IoT devices typically lack an easy interface, require an installation process, and don’t give many warnings before they stop functioning.

Being endpoints for services and requiring constant connectivity to be useful, IoT devices are susceptible to connection or hardware issues that could render them paperweights. It’s the imperative of a device maker to ensure that their product continues to delight customers long after installation.

The 2 Ways Devices Die

There are plenty of ways for IoT devices to die but they usually happen at the end of one of two cycles: duty cycle or usefulness cycle. The latter is the equivalent of death by old age. At some point, even the best of Internet services stops being useful or we grow tired and devices stop getting used and they get unplugged. Device nirvana is when it remains plugged in and useful (see the oldest webcam still connected to the Internet).

However, a duty cycle might end before the usefulness of the product ends. For IoT devices, the causes of failure are either hardware or software related. Devices on their way to dying usually exhibit a few symptoms of sickness before their demise.

For devices that actuate, such as switches, plugs, lights, and thermostats, these can take the form of higher than usual latency, a partial successful command (e.g. the thermostat turns up the heat, but not by the requested temperature), a lack of confirmation after success, or going to the wrong setting altogether.

For voice-first devices, sickness can take the form of slow or slurred speech, incorrect responses, or an inability to understand the user.

For WiFi-connected speakers, there’s usually crackles and pops or artifacts in the playback, such as static or humming.

Generally for all IoT devices, over time, the end of a duty cycle might be seen through flickering or nonfunctional LEDs, wrong coloration of LEDs on wake up, a physical discoloration of the device, and inconsistent or unstable connectivity.

The Causes of Device Death

Bringing a new IoT device to the market comes with many obligations. It instantly ties you to a roadmap in several areas where you’ll need to constantly fix, tune, and add features as people touch your product for the first time.

Without these improvements, the products will reach the end of their duty cycle earlier. Of the causes of death of an IoT device; it’s either from a malfunction of the service to which it’s supposed to connect, from a breakdown in the software running local to the device, or from the device’s physical degradation.

The I in IoT means that a device has a service behind it. IoT services are designed for remote actuation of the device, the collection of sensor data, or both. As supporting services evolve, servers need to be updated. If the business behind the IoT service fails, gets acquired, or changes priorities, the service can be sunset and if the IoT device is reliant on this service, it will become useless.

Software on IoT devices needs to be constantly monitored and updated for changes that could brick the device. Depending on the device, this could include both firmware or applications running on a host processor. When we were working on the Ubi, the firmware controlled the LEDs and sensors on the devices and also had a watchdog to reset the device if the Android host processor failed.

The voice interaction was built off of Google and Android speech recognition software. As Google made updates to this service and required updates to Android to support, we had to continually push new updates to allow the device continue to function. It was a constant chase.

Any IoT device that’s running applications on an OS that requires updates is asking for trouble, especially if the device’s operation relies on the OS being registered, like Android for Google Play services. At some point, services can be deprecated and brick the device. In screen-based devices, this could take the form of users having to manually update settings or apps that stop functioning.

The cause of device death that’s most difficult to manage is when components physically break down. In a duty cycle, this can come from basic wear and tear, heat, or from other mechanical breakdowns. Plugs and connectors have a finite limit to how many times cables can be plugged and unplugged and batteries can only be recharged before they become lazy. Heat also leads to premature aging of a device.

With the Ubi, we experienced the adhesives starting to breakdown, releasing antennae from their proper positions, or solder weakening causing device resets. Devices that have moving mechanical parts will also breakdown without maintenance. Friction over time can eventually break anything.

Why IoT Devices Are More Susceptible to Death

IoT devices are particularly susceptible to breakdown and death for a number of reasons. First, IoT is a new category of products and, compared to appliances or vehicles, has many fewer generations of lessons learned. Many startups are involved in IoT and to top off the general inexperience of the industry, these startups are also first time manufacturers of consumer goods and need to learn the hard lessons of quality assurance and testing.

Another major driver of IoT device death is the usage patterns. Typically, these devices are always on, either awaiting receipt of a command or streaming sensor data. While non IoT devices might go into sleep mode, IoT devices such as webcams are constantly on, generating heat that breaks down components.

Also, devices might be tucked away in hidden locations where heat gets trapped. Many times, the device will fail without displaying any symptoms of illness. It is even worse when the device is professionally installed, meaning that it’s death requires a professional replacement.

The infrastructure behind IoT devices also needs constant support and updates. Many first time device companies might not factor in the cost and effort required to keep the services alive and eventually shutter the services. On top of this, security protocols change or new vulnerabilities need to be patched and device makers need to be constantly vigilant that their device isn’t used to exploit users.

What Can Device Makers Do?

Device death prevention should be baked into the design phase of the product. Product managers can provide a target for the longevity of the product and then designers can build this into the design and the BOM selection.

Testing the device to prevent death is also critical. This includes running a burn in after manufacturing (leaving the device on and running through many cycles of tests before shipping), testing in extreme environments (high/low humidity and temperatures), and failure testing.

Early warning systems to alert of problems are also helpful. This could include monitoring the internal temperature of a device, measuring latency of actuation, or just alerting the user and the support team as soon as a device is disconnected. Likewise, a user-accessible portal can help communicate issues with a device and provide diagnostic and repair tools.

For any IoT product, it’s table stakes to have over the air updates to quickly mitigate any issues with software or firmware that could come up. For devices that require installation, having replaceable modules, such as a compute module, that can be removed by a user, can lead to a better user experience. We can think about what a 10/10 iFixit rating might look like and work backwards from there during the product design phase.

Dealing with the Inevitable

Some devices’ duty cycles might last much longer than their usefulness cycle. However, if they’re devices that we love, they probably will reach their demise too early. Makers can plan for this in a few ways:

Provide a replacement path for users who love your product
State upfront the expected lifespan
Discount on an upgrade to the next generation of device
Support a community of people who can repair or refurbish your product for other users
Monitor and report the health of not just a single user’s device but all devices and the health of the system
Send alerts if something isn’t working well before a user is aware of the issue

If a device becomes obsolete, there’s also an opportunity to repurpose the device. What if it had some Easter Egg inside? What if the Amazon Echo could be screwed open and turned into a flower vase?

IoT device makers can embrace the reality of the blue ring of death and plan for it. All devices will eventually fail. No device is online 100% of the time. No device will be useful forever. Since IoT devices are becoming a regular part of our life at home and at work, we should plan for them to remain more permanent fixtures.