IoT Learning Algorithms and Predictive Maintenance — Part II: IoT Architecture

Author: Dr. Taşkın Deniz, Data Scientist / Consultant @Record Evolution GmbH

Record Evolution

Published in

IoT & Data Science

23 min readJan 4, 2019

Summary

The article tackles smart data processing of the Internet of Things (IoT) in a predictive maintenance context and relates this to recent developments in semi-supervised learning. While written with an eye towards a non-expert audience, the article references recent scientific publications. We leave it to the curious and technically oriented reader to expand their knowledge on the ideas we have sketched out (see References). We aim to be informative and open minds to stimulating discussions on IoT data analytics.

We cover the topic of IoT Learning Algorithms and Predictive Maintenance in a series of three articles. In PART I, we present a simple case study and discuss some learning algorithms related to it. In PART II (the current article), we focus on IoT data analytics and applications to IoT design for predictive maintenance. In PART III, we review recent literature on semi-supervised learning and compare the foundations of different methods. We introduce real-life cases in which few-shot learning may provide an efficient technique for smart IoT analytics and data streaming. At the end of this article series, you can find a glossary of terms and suggestions for further reading.

We don’t claim to be exhaustive — IoT is a vast and flourishing topic. Feel free to comment and start a discussion with us.

Prologue

We concluded PART I of this article series with several open questions regarding IoT data analytics and the corresponding machine learning algorithms:

What are the properties of different IoT data sources?
What is the structure of different resources in the IoT hierarchy?
How can the data be stored and in what format?
What are the recent developments in machine learning with respect to the IoT problem?
How do different algorithms compare? What are the open problems therein?

In the present article, we will touch on the first three questions, leaving the rest to PART III.

1. Introduction to the IoT Data Infrastructure

IoT is the buzzword that references interconnected devices embedded in a global network. Namely, IoT is a Wireless Sensor Network (WSN) of devices employing one or several communication protocols (e.g. WAMP [28], MQTT [33]). As such, IoT can exist in various contexts such as smart-city, urban planning, agriculture, industrial production, autonomous driving, maintenance, and more. It is said that we will be surrounded by about 17 billion devices potentially streaming data into the cloud by 2020, meaning roughly 5 quintillions (5x10ˆ18) bytes of data produced on average every day [34]. As exciting and promising as it sounds in this early development stage, IoT brings along several challenges. This article addresses some of these challenges.

In PART I of this series, we presented a simple thought experiment around predictive maintenance and IoT. We suggested that we should filter streaming at the spot (meaning, for example, at a factory shop-floor or any corresponding locality) via running intelligent models at the edge of the IoT network. This was presented as a dynamically selected filter parameter taking into account local changes (e.g. as simple as an open vs. closed window in two identical factories). As easy as it sounds, such a process requires, first of all, appropriate learning algorithms (PART III) that can deal with data imbalance and multi-modality. Moreover, the process requires an architecture to build and distribute intelligence: secure messaging, data storage, and app deployment infrastructure. That’s why, first and foremost, one should have a deep look into the corresponding IoT infrastructure and software ecosystem. In the following sections, we will explain what we mean by that. Moreover, we will look closely into the relevant technicalities and answer why the problems presented in PART I make sense.

The core of our argument is that to deal with large data streams, a network of local devices in a distributed computing framework can be formed. Imagine several nodes (such as cameras, temperature or pressure sensors, counters) sitting at context-relevant locations or simply attaining mobility in an area, e.g. in a factory shop floor. These devices are built to measure physical observables and stream data to a server. Until recently, this was implemented straightforwardly (e.g. in a master/slave model). Alternatively, the devices can potentially communicate with their peers and evaluate the data. Further still, they can form data stream packages (local buffers) and also act as local servers. The crucial point is that, ideally, these devices should pre-process the data they absorb before streaming the data to a host server.

As an end product, the Big Data will be cooked into an AI (artificial intelligence) model using immense computational resources (that possibly live in the cloud or at a ‘nearby’ computation center). This is also an issue we won’t address at this point. Finally, the central AI will be inherited by local devices recursively to improve solutions to real-world problems. How would one approach this issue in a bottom-up scenario then? Let’s start with the properties of IoT data.

Figure 1. The IoT data collection and streaming brings along several challenges [13]

1.1. IoT Data Characteristics

First of all, we need to understand the general properties of the data picked and published by sensors in a general IoT setting. IoT data is not just another kind of Big Data. To start with, one needs to be aware of the following six issues or the so-called “challenging 6 Vs” of IoT [8].

The first three Vs are general:

Volume (of measured data)
Variety (of data sources or modalities)
Velocity (of data generation)

The remaining three Vs are often context-dependent:

Veracity (or, in other words, the credibility of the data)
Variability (in streaming parameters)
Value (for the business analytics)

Some of these properties may become more important for a given problem. For instance, in an industrial setting, it is not always the size of the data that matters the most. Sometimes network security and business value step forward. Think of the data from product-counter sensors with a sophisticated image recognition algorithm for a production period of 10 seconds. In general, there can be much faster processes, e.g. polymer production or metal press; bandwidth is adaptable to a large extent. We assume that one can restrict the volume of data from growing — in case it is useless in a given context. This task will be optimally executed if smart filters are implemented. This is probably the simplest concern of a data scientist while building an IoT platform.

Figure 2. Figure adapted from a review article [13].

For one, all these main properties combined, many issues stem from current IoT paradigms, because “… the abstraction of IoT data is low, that is, the data that comes from different resources in IoT are mostly of raw data and not sufficient enough for analysis” [13]. Consequently, simply pooling all the raw data in a server and hoping to discover insights in this so-called data lake seems fruitless. This is not only because of technical constraints such as computational resources but also because of time and space correlations ignored by constant (filterless) streaming and pooling.

Second, some small undetected changes in an IoT environment can occasionally claim to be of immense importance. For example, in a factory automation context, maintenance teams are supposed to know first and foremost which machine to fix when and where. Besides, the history of particular machines matters as well, which implies that different kinds of data may be collected from a specific machine and the model should be slightly adapted to it. Thus, the inheritance of intelligence to the edge of the network is indeed crucial, as well as local adaptation to small changes. On the one hand, a local extension of the model is needed for an “on-the-spot” data evaluation similar to what we suggested before (should give ultimate importance to the precursors of a breakdown). On the other hand, the accuracy may suffer from undetected local noise sources (as simple as a factory door being open while this is undetected at first). We will discuss this issue and several techniques therein in the following sections, and then in detail in PART III.

Third, with or without the issues mentioned above, IoT requires a database and huge processing power for training a general model (imagine the success of a Google speech recognition model vs. a custom solution). At the end of the day, we would like to achieve both features. Namely, the generality of a central AI model evolving in a slow time scale and the intelligence running locally for evaluation and data streaming. The devil lies in a subtle trade-off between levels (up or down in Figure 2) which means we should implement AI at any level. A truly unpleasant alternative is waking up on a fine Sunday in the middle of a huge data lake, possibly including hundreds of terabytes of 10kHz-sampled sound files, although they contain no anomalies whatsoever. This already costs storage and computational resources — only to find out that they are useless.

To sum up the discussion at this point, to set up an efficient IoT environment, one has to deal with IoT-related technical issues beyond static Big Data (see article [8]):

Large-scale Streaming Data: A myriad of data-capturing devices are distributed and deployed for IoT applications, and generate streams of data continuously. This leads to a huge volume of continuous data.
Heterogeneity: Various IoT data acquisition devices gather different information resulting in data heterogeneity.
Time and Space Correlation: In most IoT applications, sensor devices are attached to a specific location, and thus have a location and time-stamp for each of the data items.
High-noise Data: Due to the tiny pieces of data in IoT applications, data may be subject to errors and noise during acquisition and transmission.

To interfere at any level/layer, one has to run smart algorithms embedded within IoT applications [13]. This is easier said than done. To accomplish this task, recently semi-supervised learning algorithms are employed to help balance missing samples or even creating more samples (with GANs [1]), if necessary. These techniques, on the one hand, can “model the small amount of labeled data with a large amount of unlabelled data” [13] (transductive learning). On the other hand, they can incorporate a “small amount of unlabeled data with a large amount of labeled data” [13] (inductive learning). We assume, in this series of articles, that we are dealing mainly with an inductive learning problem.

1.2. IoT Data Analytics and Machine Learning

IoT data is not only Big Data but it encapsulates multi-modal observations (e.g. audio, visual, and numeric) and its parameters may vary in time in a given context. These properties make it tricky to adapt data analytics/artificial intelligence to IoT [13]. IoT simply needs smart algorithms that are flexible in including various sources and changing properties as well as an update infrastructure to attain the most intelligent model available (e.g. from a Cloud registry). Most of all, in practice, we need to transfer previously learned categories and relations to make sense of novel observations.

The transfer of already learned classes and relations is crucial for an efficient learning strategy. Humans use this strategy to learn new behavior based on similar learned behavior or remembering features of new objects based on known classes of objects. Of course, we are not interested in implementations of few-shot learning or in general-terms transfer learning only to mimic humans via smart IoT devices. This technique is per se important for the efficiency of the vast IoT framework in preselecting the data at the source.

In PART III of this article series, we review a set of models and semi-supervised learning techniques that address these issues to a certain extent. In particular, special attention is paid to current developments in Deep Neural Networks (DNN) and Support Vector Machines (SVMs). Other machine learning models are also mentioned for the sake of completeness.

Figure 3. Predictive Maintenance with IoT [22]

Machine learning is crucial for IoT but we take a step back at this point to write about industrial IoT infrastructure.

1.3. A Note on Messaging and Security

Building smart apps distributed to different devices requires more sophisticated architecture than simple monolithic software. Thus, we enter the realm of microservices and infrastructure-managed versions and messaging.

Micro-services: Security, Stability, and Data Management

In this section, we deal with messaging and app deployment technologies. These are important as part of collecting data and later distributing the updated AI to the edge of the network. These two topics are covered in the same section because WAMP technology encapsulates both Pub/Sub (publish and subscribe) and RPCs (remote procedure calls). This makes deployment and data streaming within the same protocol simultaneously possible. Telecommunication protocols and messaging technologies are motivated, on the one hand, by compatibility issues between various components of the assembly lines. On the other hand, we have security and stability issues related to intrusion prevention and network downtimes. Besides, the complexity in networks of apps extended in space and time calls for optimized data-binding mechanisms. One additional challenge is that the corresponding network is dynamic, meaning that it is consisting of nodes or connections being activated and deactivated as a part of an automated production process.

Restricting our scope to industrial applications, below we sketch out an overview of messaging technologies and issues therein:

OPC (Server)

OPC (Open Platform Communications) was introduced in 1996 as a unifying protocol for industrial automation. The main idea behind it is that one assembly line can work with many different control devices e.g. PLCs (Programmable Logic Controllers) which may have incompatible messaging protocols among them. An OPC server simply collects data from all PLCs and HMIs (Human Machine Interfaces) in a factory environment using a server-client model. Traditionally, PLCs used different Modbus protocols and ethernet networks. Given the compatibility issues, an alternative to OPC was custom software built for a given production line, which is time-consuming, expensive, and inflexible. On the one hand, this disadvantage justifies a unified protocol (OPC). On the other hand, the disadvantage is that OPC is platform-dependent ( Windows-based) and not secure (presenting open ports).

OPC UA (Server)

The aforementioned OPC protocol was the ground-setting telecommunication standard in factory automation until 2006 when its new version OPC UA (Open Platform Communications Unified Architecture) took over. As such, OPC UA surpasses the capabilities of OPC. To start with, OPC UA is platform-independent whereas OPC’s ecosystem resides in the Windows operating system. Besides, there are several open-source OPC UA packages available in C, Python, JavaScript, etc.

The structure of an OPC UA server is simple. It has a root node that opens a communication channel (which can be secured by firewalls) and address space, in which every node is an object (in an object-oriented programming sense) with corresponding functions and attributes (static and dynamic data). The root node can be connected to the outer world with a secure protocol such as MQTT and/or WAMP.

Today, OPC UA is seen as a gateway to the so-called Industry 4.0. The global plan to attain a fully automated industry is to embed OPC UA technology not only in new generations of control devices (PLCs, HMIs, etc.) but also in any device that connects to industrial IoT networks. This simplifies the software-related manual work that has to be managed and paid for. However, OPC UA is a machine-to-machine protocol while a fully automated setting often has to interact with the outer world meaning to a host server or an app registry.

The application ecosystems can be in the cloud or your local facilities remote from the factory shop floor. Consequently, the information to be transferred from/to the local IoT network to/from the “outer” world or updates of applications should be managed. To accomplish such data flow, there are several industry secure messaging technologies, of which we will cover only MQTT and WAMP [28, 33].

Secure IoT Messaging: MQTT and WAMP

MQTT is a secure IoT communication protocol with a pub/sub mechanism. There are many router implementations available, the most prominent being RabbitMQ maintained by pivotal. MQTT has been around for some time now and is used widely in the industry. Using a messaging solution, you find yourself quickly in a microservice environment with different types of client devices and client-side programming languages. Each language has its own data structures that you want to receive or send over a messaging service such as MQTT or WAMP. MQTT leaves it to each programmer to encode and decode the message payload on the sender/receiver side since MQTT can only transmit byte streams. The process of encoding structured data to a byte stream is called serialization. A disadvantage of this approach is that the network has to agree on a serialization format to enable any receiver to understand the message content. Google’s protobuf serialization framework with compilers for different languages is one of many ways to handle serialization with MQTT. Javascript, however, is not supported, so it cannot be used to communicate with Web browsers.
The WAMP protocol, too, provides the pub/sub mechanism as MQTT does but also features some important additional features. One of them is the transparent support for a diverse range of serialization protocols (CBOR, UBJSON, MSGPACK, …). Using the WAMP client libraries (autobahn) and a WAMP router (crossbar.io), you can send data structures from any client to another, each sending or receiving the payload in their native format. For example, you can send a dict from a Python client and receive a JavaScript Object in a JavaScript client.
An additional important feature is the possibility to use the second major communication pattern, “Remote Procedure Call”. One important consequence of using a messaging service combining both patterns is that you need not rely on two different technologies for pub/sub and RPC to implement full application architecture. (Often REST-based web request handling is used to feature RPC-style communication, which requires a web server as an additional architectural node next to the message router.)

There are OPC UA — MQTT and OPC UA — WAMP and MQTT — WAMP bridges available. The network architecture is flexible and can serve the demands of heterogeneous industrial IoT environments.

2. Industrial IoT Architectural Design

Predictive Maintenance and IoT go hand in hand. To cite [21]:

Industrial IoT is making the transformation to continuous condition-based monitoring and predictive maintenance easy and affordable. Four key trends are driving this change:

Wireless Connectivity
Inexpensive Sensors
Cloud Computing
Analytics or Artificial Intelligence (AI)

While in isolation, each of these technologies brings some value. When combined into a single solution, they have the power to transform the industrial world. [21]

An IoT architecture consists of four major parts:

> A. Edge Devices

The name is self-explanatory. Edge devices reside in the far end of an IoT network, close to the data source. They are simply consisting of a processing unit (an ARM architecture or even a smaller computer) plus one or more observing endpoints (sensors, cameras, microphones, etc.).

As mentioned, there are two basic pyramids of IoT tasks: the pyramid of data streaming in real time and the inverted pyramid of data management and computation. The closer one gets to the edge, the smaller one’s computational resources will be in general. Edge devices can only implement a modest amount of pre-processing. Thus, we need to choose our methods wisely. Our point of view is that, although this rule of thumb is valid, we may need to bend the rule for accuracy, efficiency, or interpretability reasons.

> B. The Network

A network consists of connections and nodes. Such connections and nodes may be overly heterogeneous as an internet network is highly dynamic. This means that new devices can join or leave the network or such devices can demand a new protocol. This complexity should be managed by a flexible IoT architecture.

> C. A Device Management Service

An IoT network contains heterogeneous resources and tasks. Task management in a device means that we deploy and employ an appropriate app therein, allocate resources to its functions, and keep the device meta-data to maintain the device status. This will be discussed in detail below.

> D. A Data Management Service

When all three aspects are there, there is still a need for a streaming and storage model. Moreover, we need a transparent data process screening. For example, data sources, following SQL queries or data encoding mechanisms, should be transparent. This topic will be presented in detail below.

…

In the following subsections, we focus on details of the last two aspects, because IoT hardware (sensors, CPUs, memories, etc.) is a vast topic and tackling it will not contribute much to our discussion at this point. Another reason regarding the network topology is that it can vary immensely subject to the needs of the user. By topology, we mean which devices are connected using various protocols and bridges between protocols, etc.

…

2.3. How to Manage Your Devices

In this section, we mention the challenges in device management. Let’s start with the heterogeneity of environments and configurations (brownfield), deploying over the air (wireless), remote managing connectivity settings, network quality (buffering of data), security, etc. are major issues.

Describe the challenges with:

Managing devices: IoT devices themselves must be observed and device settings must be managed according to the needs of the IoT environment.
Heterogeneity of environments and configurations (brownfield): Any new software architecture must take into account and coexist with live software already in use.
Deploying over the air: When you are ready to distribute an application over the air to a group of devices, you create a deployment for the application. A deployment delivers software over the air to the application environment in devices on which it runs. Only one deployment is active for a particular device at any given time.
Remote managing connectivity settings: With the same mechanism as device management, one can also manage connectivity settings (with various protocols).
Network quality (buffering of data) is not always guaranteed to be top-notch. Thus, we must work with local buffers if we don’t want to use data. This also involves an optimal streaming model. The implementation of such models requires automated parameter selection.
Network security is of utmost importance, especially against intrusions or attacks. We have already mentioned the security issues above (look at our discussion on OPC, MQTT, and WAMP).

In an industrial environment where IoT is embedded as part of the production process, one has to deal with various data sources. This process is dynamic and automated, and in general, any IoT deployment or update must not interrupt the production. First of all, functionality in applications may evolve or a deployment mechanism must be available in case a new functionality is to be built. Moreover, new devices can be added to the network. This should be managed in a modular way so that evolving applications can match the needs of this complex dynamic environment. The management method should be flexible enough to implement the app registry locally or in the cloud. It should address one of the main concerns of such an infrastructure, that is, security: intrusion and intervention prevention.

2.4. How to Manage IoT Data

In this section, we mention some major challenges of IoT data management. To enlist: data volume, time sensitivity real-time vs. batch processing, heterogeneity (no data structures standards), data flow controls, metadata management, data quality, transform for usability, create large data histories, auditability, etc. are all issues when dealing with IoT data.

Below we describe the major challenges:

Data volume: Soon we will deal with IoT on a large scale, meaning we need an optimized storage infrastructure for this so-called Big Data.
Time sensitivity real-time vs. batch processing: We need to organize incoming data at the storage facility in real time. The alternative is batch processing which simply takes more time and resources.
Heterogeneity (no data structures standards): Data can be collected and communicated using different protocols and standards. We need to encode data at the source before streaming and decode it at the storage facility. This can also be done via WAMP or MQTT-based routing mechanisms (e.g. Crossbar).
Data flow controls: Because of this complexity, we need to keep track of all data transformations. There are several different ways of this, such as SQL code or graphical pipe representations.
Meta-data management is crucial when dealing with network health and possible streaming optimization. We also need to keep track of the properties of data sources such as machines, the factory environment, device data, etc.
Data quality, transform for usability: We need to deal with the missing data at the storage facility as well as the data source; quality management must be automated and should be transparent.
Creating large data histories means that we need to keep track of time series as a serial correlation in slow time scales can be crucial in predictive maintenance.
Data auditability is a crucial part of the process as data often has business value or is collected to solve a problem. It is often easier to deal with the storage mechanism if you already have a good predictive model available.

As long as data acquisition apps are deployed in devices, we can start collecting data. Nevertheless, the question remains as to how one can manage such data with immense volume and multi-modal nature. Part of this can be managed by the application deployment mechanism via RPCs, e.g. encapsulating different data structures, encoding, and decoding. Again, by RPCs streaming, parameters can be adapted by hand or in an automatized fashion. However, we need to store the data consistently, meaning sources should be respected and table operations (SQL) should be transparent. Moreover, time-scale data brings along even more challenges as often data has serial and parallel correlations — as often physical proximity can be a crucial aspect. That’s why one needs to keep track of bathes and device meta-data in a dynamic environment. How can one manage all this complexity? Let us answer this question.

2.5. The Record Evolution IoT Development Studio

Once we have sketched out the main problems concerning device and data management, we have come to present a solution. Our solution is implemented in a new IoT device/application management platform called the Record Evolution IoT Development Studio. The IoT studio offers a WAMP-based swarm management environment. The complexity of the architecture addressed by our solution is presented in the flowchart below:

The Record Evolution IoT Development Studio deals with the issues mentioned above using WAMP technology and the IoT-relevant application ecosystem

Apps can reside in the cloud or a local registry while app rollouts can be easily implemented using the functionalities (RPC-based) of the IoT development studio. Communication is secure and consistent (pub/sub-based). This is a snapshot of a running instance of the IoT development studio’s frontend with all its buttons involving app management. Using the development studio, your app can be modified:

A snapshot of the IoT development studio’s frontend

Going into the technicalities of the Record Evolution IoT Development Studio is beyond the scope of this article. In sum, as long as a recursive learning inheritance mechanism (so-called online learning) is available, one can deploy new versions of intelligent models to various devices via container technology (e.g. Docker) and WAMP (e.g. Crossbar) messaging.

2.6. Choosing the Storage Architecture: This is a big topic on its own. The first question is: which data model shall I use? The second question is: which flavor of SQL/NoSQL shall I go for and what storage infrastructure should I choose? Choosing proper storage technology for a long-term data management solution is no easy task as it involves many technical and conceptual decisions. If you want to build the solution on your own using open-source components, you can look at diverse time-series databases such as influxDB or PostgreSQL with timescaleDB extension, or you can choose a cluster storage solution such as HDFS, GlusterFS, or Cassandra. Each has its advantages and disadvantages.

2.7. How to Manage Your Data Processes: Once the data is stored, you need tools to transform and analyze the data to do something useful with it. This could be classical analytics or heavy computation training machine learning models. If you have processes to enter, transform and extract information then again you need more logic to control these processes. You need data governance, data quality, and automation logic. Tools that provide this kind of service need to be hosted and administered on-premise or in the cloud in addition to the raw storage technology.

To simplify these tasks, we offer the Record Evolution data science studio. For a discussion on the topic and this platform, we refer you to the following medium article [36].

**WAMP-based IoT connector in the data science studio**

Conclusion — Part II

We are on the verge of a fully automated industry. Neither process optimization nor predictive maintenance can be possibly thought of without an IoT infrastructure. Motivated by this, we have presented an IoT infrastructure and software ecosystem focusing on industrial applications. Keeping in mind our main aim, predictive maintenance, and process optimization in industry, we have sketched out some important topics in industrial IoT. Also, the current article has addressed some of the questions posed in PART I. These include:

What are the properties of different IoT data sources?
What is the structure of different resources in the IoT hierarchy?
How can the data be stored and in what format?

IoT is the future but IoT data poses numerous technical challenges. In our opinion, collecting and streaming huge amounts of data driven by AI is one of the most pressing issues therein. In theory, IoT can be consisting of interconnected and collaborating nodes instead of simple constant streaming or a master/slave model. In practice, accomplishing this requires a messaging and deployment infrastructure. As a possible solution, we propose the Record Evolution IoT Development Studio, a WAMP-based management and development environment. Although not tested on a large scale yet, the IoT development studio is theoretically scalable and offers easy-to-use functionality to manage devices and apps in factory environments.

IoT data properties and data storage infrastructures are also mentioned above. Major problems inherent here include keeping track of correlated data sources, collecting meta-data, and keeping time-stamps consistently. Moreover, a transparent data warehousing method helps us clean and prepare data for the training of large-scale network models. For these tasks, we propose the Record Evolution Data Science Studio. You get a cloud data platform that has IoT connectors and SQL functions presented as data pipelines. The platform provides users with an API function and custom reporting features (e.g. D3, HTML). Visit our channel for more information on the cloud data platform and stay tuned for more.

…

We are still in debt to the reader and we will atone for this in PART III by answering the following questions:

Why are machine learning algorithms relevant to IoT edge computing?
How are these algorithms realized?

________o_______________o_______

Acknowledgments

Thanks to Dr. Marko Petzold and Dr. Sven Dorosz for our discussions. Special thanks to Dr. Petzold for his contribution to the data and device management sections.