IoT System Tests :: Checking for Failure
Internet of Things sensors use a diverse range of technologies to collect information, communicate with each other, and transmit data to computer systems that store and process the data.
In a typical application, an IoT device wakes up at regular intervals, takes measurements, and makes decisions based on a combination of sensor data. It also communicates with other devices and may send data to back-end systems in the cloud for processing.
For example, a medical tracking device may use different types of sensors to monitor various parameters in the human body. The system can then process the data internally or through a cloud-based system, generate reports and send relevant information to a medical officer or physician.
The physicians can then use the captured data to initiate remedial actions such as a drug intake. And this action can be triggered and administered remotely from a computer or mobile device connected to the medical device through the internet. Some systems may even include advanced applications to monitor fluid levels in the patient’s body, and automatically initiate transfer if below a set threshold.
Due to the mix of different devices, platforms, protocols, and communication gateways, IoT systems often suffer from potential performance, reliability, interoperability, and scalability problems.
Unfortunately, testing the performance and reliability of each component can be challenging.
Problems affect the smooth flow of processes within the IoT system: in this article, we’ll be looking at testing methods.
What to Test in IoT and Why
With the IoT market growing at a rapid rate, manufacturers are rushing to produce a wide variety of low-cost sensors. To keep costs down, some of them may overlook or sacrifice some functions. This makes the devices vulnerable to multiple security, performance, and reliability issues.
Typical limitations of IoT sensors include small physical sizes, low power sources, small amounts of memory, and limited processing power. This means minimizing the number of processes, and consequently, the size of the applications. As designers struggle to cope with these limitations, they may leave out crucial processes and expose both the sensors and the devices to a wide range of vulnerabilities.
Benefits of Testing Internet of Things Devices
To provide value to customers, it is essential to test devices and networks to verify that they are working as per specifications. This allows developers to identify and address issues or bugs that could affect performance, reliability, or user experience.
In addition, testing ensures compliance with safety and other standard regulations. For example, a medical implant must be approved by FDA before entering the USA market. In particular, the battery must meet performance and safety standards as set by regulatory bodies such as UL (Underwriters Laboratories). Other regulations ensure that the devices do not have excessive emission levels.
Failure to observe regulations can create more problems for users and manufacturers. For example, in the year 2017, the FDA had to recall about half a million St. Jude Medical’s RF-enabled pacemakers after identifying security flaws in the medical devices.
A test showed that hackers could access the device remotely and reprogram it to provide inappropriate pacing or even run down its battery, hence risking the lives of the users. Although FDA did not recommend the removal of the device from the market, it approved a firmware update that addressed the vulnerability and users were requested to visit their health providers for patching.
Consider these other benefits as well:
· Identifying flaws which provide an opportunity to address performance, interoperability, security, and safety issues. For example, security flaws such as sending unencrypted data allow criminals to intercept and access sensitive private information. Such access will enable hackers to view live-feeds from surveillance cameras and can be risky since this information can be sold to a competitor.
· Accelerating the time-to-market for products.
· Improving the user experience by ensuring seamless operations across a wide range of devices and platforms.
A comprehensive IoT testing system includes the sensor, connectivity, mobile apps, environment, and the web portal. However, test scenarios may differ according to the application or system design.
In most cases, the best approach is to understand what needs testing, why, and what to test for. Let’s take a look at which factors you should be testing in more detail.
Performance Testing
This looks at how well the devices, networks, software, internal and external applications are functioning to achieve the objective of the IoT devices. The performance testing determines a wide range of parameters, including the execution flow, timing, response times, reliability, and stability.
It also checks usability, customer satisfaction as well as how well an IoT system is meeting the SLA. In addition, testing scalability establishes how adding more devices and workloads could affect system performance, responsiveness, and stability.
Performance testing should look at three main levels:
· System level: processing, analytics, database, etc.
· Application level
· Network and gateway level: To test technologies such as Wi-Fi, Bluetooth, Z-Wave, RFID, NFC, and protocols such as HTTP, CoAP, MQTT and other IoT specific protocols
The tests can, for instance, establish the ability of the embedded software to perform the required computations or the reliability and stability of communication networks under certain operational conditions.
During performance testing, it is important to also establish conditions such as power usage, temperature changes, memory usage, and other variables that influence or are influenced by the processes.
Security Testing
Each node in an IoT system is a potential entry point for hackers and could compromise the performance of the individual devices or a set of connected equipment.
Vulnerable points include applications that collect data, sensor networks, machine to machine protocols, the middle-ware interfaces, and more.
Researchers have demonstrated several security weaknesses that allow criminals to access IoT devices and manipulate data or how the systems operate.
For instance, a hacker can intercept data from a medical device and manipulate it such that the information reaching the physician is inaccurate. The physician may then respond by increasing or reducing the dosage - thereby risking the life of the patient unknowingly.
Similarly, a hacker can alter the dosage of an automatic drug dispensing system, or deplete the battery of a device such as a pacemaker — hence expose the patients to greater risks.
In one of the tests in a security conference, researchers were able to demonstrate how criminals could hack into an IoT-based wheelchair and disable its safety features hence exposing the rider to risks.
The security vulnerabilities span across all industries, tests have also shown that how hackers can access a vehicle’s navigation data, and alter its control systems data, including taking over the steering and braking systems.
The security tests verify the security of the devices, data, applications, and associated networks. It looks at device authentication, data protection at the device level during transmission, and at the servers.
Testing vulnerability at the device and protocol levels helps identify flaws and prevent attacks at the source, while also verifying the effectiveness of encryption and decryption ensures security during data transmission.
Usability Testing
The usability test establishes whether devices are easy to use and if they are suitable for the application, easy to use and able to provide clear results. The testing looks at the end user experience, including the functionality of all features. Factors to consider include portability and the ability to capture and send notifications as well as error messages when the device encounters problems.
Depending on the device design, the test verifies its ability to collect and process information within the device or in cloud systems. It also ensures that it can display clear information on mobile devices or computers.
The usability is best tested by the users themselves. The QA teams may come up with certain queries that real users can answer about their experiences and expectations. However, the tests should also be done internally by the designers, developers, quality control, marketers, customer care and others.
The marketers and customer care will know what the customers are looking for while customer care will find out the features where users are experiencing problems.
Network and Connectivity Tests
Seamless connectivity is critical in an IoT network. It ensures all devices are always in contact with each other, and with other systems such as the IoT servers.
Network performance testing examines the connections, and data transfer speeds between devices and different network layers. This helps in identifying issues such as latency, bandwidth bottlenecks, or packet losses that can break connections between devices or networks.
The connectivity test is carried out both when the network is up and when it is offline. Here is a breakdown of how these two tests differ from one another:
· The online test looks at the connectivity between devices and applications, data transfer, and network security.
· An offline condition tests out what will happen when the network is unavailable. In critical applications, devices such as pacemakers or health monitoring devices should continue functioning regardless of the state of the network. The device must have the capacity to store or process the data it collects during the offline state and then transmit it when the network resumes.
IoT Device Compatibility Tests
This test is performed on the application and network layers. It tests the compatibility between different types and versions of hardware, software, and networks.
The tests establish if there are performance issues that arise from combining certain IoT devices, networks, mobile devices, platforms, operating systems, and environments.
Device Interoperability
This assesses how well combining a wide range of devices using different interfaces, network topologies, and protocols will work together and allow the exchange of data across platforms and devices.
The tests verify that the systems will produce results in a standard format that users and applications can access, regardless of the make and model of the user interface device, type, and version of operating systems.
Compliance and Regulations
IoT devices are required to meet several regulatory and compliance standards depending on the application. Testing ensures that all the devices are safe and able to meet acceptable EMI levels, among other standards.
This testing should find out if the device is emitting excess interference signals, and whether it can withstand external noise from others.
Batteries, just like other IoT components, must also comply with safety and security standards to ensure reliability and reduce risks of device malfunctions — which can be fatal in medical and other sensitive applications.
In particular, the battery compliance tests look at discharge and emission levels, corrosion, and performance characteristics over its entire lifespan. This also examines different load conditions, including those outside its recommended range.
Testing Challenges for IoT Systems
Testing IoT systems is usually a challenge due to the mix of standard and non-standard hardware and software technologies. Establishing the right environment and acquiring the appropriate test gear, applications, and networking becomes more challenging as the number of devices increases.
Unfortunately, the traditional test process of activating one or multiple input triggers and then validating the resulting output is not applicable in IoT systems.
Simulating all possible scenarios is complex and sometimes not practical. Factors such as unreliable networks, complex real-time scenarios, varying device states, and sensor data as well as unstable device interactions will generate an infinite combination of test scenarios.
Other challenges include third-party components and subsystems, poor quality hardware, safety, security, and privacy issues.
As an example, there are about 20,000 devices sending data per second using a wide range of protocols. Testing all the possible combinations in a scenario like this can be difficult.
This becomes even more complicated for large systems which have grown gradually over several years. In such cases, there may be aged devices that can only communicate using older protocols such as CoAP, while newer ones use HTTP, MQTT, and other modern technologies.
Let’s take a look at the other challenges testers may face.
Complex Hardware-Software Mesh
Lack of standardization has led to a mix of diverse technologies in the IoT ecosystem. These systems are likely to consist of different types and versions of hardware and software technologies which are expected to talk to one another in real or near real-time.
Components and subsystems coming from different manufacturers and third-party providers might be using incompatible technologies, which could affect the testing process, especially when there is a failure in one device. In addition, the sensors may not be of good quality to provide accurate test results.
Testing such a mesh is a complex process that requires proper planning, test equipment, facilities, expertise, and capacity that some QA teams may not be able to afford.
A Large Number of Test Scenarios
A comprehensive test should verify how the devices and systems work normally as well as scenarios with unreliable or weak networks, low battery conditions in the sensors, security issues, data packet losses, and other practical scenarios.
This is a complex process when testing thousands or millions of devices and applications across different networks. It is also difficult and expensive to replicate some dynamic environments similar to where the IoT devices will operate in a real-world application.
Simulating such real-world load conditions is almost impossible due to the wide range of devices using different technologies. There are also unpredictable usage conditions that testers may not be aware of, or simulate, making them impossible to test. This includes the number of concurrent users, reliability of user devices and internet connections, and more.
Another challenge is determining the battery current consumption in different operating modes that range from deep sleep to full operation states.
Scalability Issues
Generally, the number of IoT sensors grows exponentially. Although the initial implementation starts with a few devices, it eventually grows to thousands or into millions after several years.
As the number increases, the current test tools and scripts, designed for thousands of devices, become out-dated and unable to handle such large-scale deployments.
Further, preparing the run test cases and analyzing results becomes more difficult, slow, expensive and prone to errors such that pinpointing a malfunctioning code or device is a nightmare.
Different Communication Protocols
The IoT devices use a wide variety of protocols for connectivity but do not have a standard set. Generally, the protocols include HTTP, MQTT, CoAP, XMPP, DDS, AMQP, AllJoyn, IoTivity, and many more. These are usually different and some do not follow a universal standard.
While some are fully developed, others as still evolving and often change as they mature. Due to the dynamic nature of IoT, it becomes a challenge with older performance testing tools since they cannot support the new technologies and non-standard or continuously evolving protocols.
Geographical and Network Challenges
Thousands or millions of IoT devices spread across different continents use a wide variety of networks to connect to one another and to communicate with mobile devices or IoT servers. Simulating such conditions when testing connectivity between sensors in different parts of the world can be a challenge.
Generally, the mix of multiple network technologies, like Wi-Fi, Bluetooth, Z-Wave, RFID, NFC, LoRa, Sigfox, and others raise concerns. When you factor in different device states, this results in so many possible combinations that it’s impossible to replicate them all.
Establishing a practical environment is usually not possible for all applications. There are many factors that may affect communications. Internal and external wireless signal levels vary, and there are other risks such as EMI.
In addition, interference or noise in the power lines or from the wireless signals will affect the quality of the communications. These are dynamic scenarios that a tester may not simulate properly.
Additionally, most IoT devices have limited resources. For example, to prolong battery life, the devices cannot transmit data frequently and are sometimes unreachable due to low duty-cycles.
Furthermore, communication relies on IoT specific communication protocols without a universal reach as well as different standard and non-standard data encoding languages.
Real-Time Testing Challenges
Other than ensuring proper operations, testing for safety and compliance is mandatory in almost all the devices. Obtaining real-time data is usually a challenge in some applications such as medical devices.
Although testers can simulate some of the conditions, it may not be possible to get other dynamic conditions that could vary depending on a human body’s temperature, stress levels, health and other factors that may influence the performance of embedded devices.
Sometimes, the equipment to be tested may not be available or not exhibit the conditions required to activate the sensors during the testing process. In addition, it may be difficult to coordinate and organize different teams and individuals carrying out the tests.
Different User Interfaces
There are different platforms for IoT devices and systems. This includes operating systems such as Windows, Linux, iOS, Android, and others. In addition, different device models and software versions may not respond the same way to certain conditions and environments.
Testing the IoT devices on all possible platforms and conditions is practically impossible. Testers may not have access to all the types of mobile devices or versions of operating systems, or the ability to test specific situations.
Another challenge is recording information from devices such as those where testers cannot change the settings to allow them to send information to a different location.
Addressing IoT Test Challenges
In addition to employing best practices, an effective test framework must address the three core components of IoT. It should have the ability to assess the devices and applications, data processing and analysis, as well as communications.
Having a comprehensive test strategy covering all aspects of IoT can go a long way in ensuring stability, reliability, and dependability. An IoT test framework needs to overcome most of the challenges by having as many options and flexibility to accommodate a wider range of test scenarios. It is important to have the appropriate test equipment and tools that can ideally check a number of related parameters simultaneously.
For example, using a mixed domain oscilloscope, you can check both the analog and digital signals and see the relationship between them and other conditions such as power levels and commands.
The mixed domain oscilloscope captures the RF, analog, and digital signals simultaneously and then time correlates them to determine if there is a specific or common problem that affects one or all of them.
For example, the oscilloscope can identify an issue with the radio signal and also show if this is due to wrong control bus commands, power, or any other fault- hence making it easier to isolate the problem.
Basically, the features of an effective testing solution include:
· A test lab where testers can simulate real-time experiences and also experiment with a wide range of combinations.
· Protocol simulators with the support for most of the commonly used devices.
· Data recording technologies that work across a wide range of devices and applications
· A virtualized test environment to automatically and quickly respond to changing real-time requirements.
Internet of Things testing solutions
There are different types of hardware and software test tools for IoT devices and networks. The choice depends on a range of factors, including application, environment, and parameter to test among many other things.
Some providers such as Keysight, Infosys, and TransactIQ have solutions for performing a wide range of IoT tests. Most of the commercial tools support several commonly used IoT protocols and network simulations.
The Infosys, for example, supports HTTP, MQTT, AMQP, REST over HTTP, Web sockets, Kafka, and CoAP. In addition, it has the ability to test different network simulations for each protocol. Its automated scripts enable it to generate desired load or device simulation from the cloud, thus eliminating the need for a tester to acquire the hardware.
The TransactIQ’s solution performs a comprehensive analysis of network activities such as transfer speeds, latency, bandwidth, and packet losses as data moves from one network layer to another.
The solution is suitable for most of the common technologies, including WiFi, LTE cellular, Sigfox, Bluetooth LE, LoRa, NB-IoT and more. Also, TransactIQ provides other solutions such as testing security on non-HTTP protocols like CoAP, MQTT, and others.
Conclusion
Users expect all connected devices to work and communicate flawlessly, including when they upgrade, update, or add new devices to the network. For this reason, manufacturers and service providers need to focus on quality and reliability which can only be verified through testing.
With a wide range of technologies that are constantly changing, it is not possible to pinpoint a single solution for testing an entire IoT ecosystem. However, QA teams, developers, and manufacturers should aim at getting test solutions that cover most of the common devices, applications, and network technologies. These should also be flexible and possess the ability to accommodate new technologies.
