GLAMPP (Grafana, LPWAN, AWS, MQTT, Python, Postgres): The Natural Tech Stack for the Industrial IoT Community

The first time I heard of LAMP, around 15 years ago, I had the 💡moment.

• Linux: The operating system — The foundation of the LAMP stack
• Apache: The HTTP Web Server — Processes requests from browsers
• MYSQL: The Database — Stores the web app data
• PHP: The software language — Implement the specific web app logic

and BOOM! you have yourself a web app.

LAMP Tech Stack

Every company has a unique tech stack which is like the skeleton of its products and services. Just like a human skeleton, the tech stack is the internal structure of the product. It comprises of multiple rigid components (bones) that support and protect the product (body). As an example, companies might decide to use Python as the underlying software language of their products (And not Java, C, JavaScript, etc..). Python is only one bone of the skeleton. The specific software written in Python that implements the functionality of the product is the muscles, joints, and tissues that wrap the bone, support it and hold everything else in place. Similarily MYSQL database is just another bone. Without defining the schema, indexes, queries and loading it with data it can’t run… However, when one of your product’s bone cracks or breaks, believe me, you would feel the pain.

“The magnificence of my skeleton is hidden by the weight of my flesh.”. Vera Caspary, Laura

Companies include in their tech stack tools, frameworks, databases, hardware, software languages, and platforms. They then illustrate the hierarchical relationship between the various components either as a stack (top-bottom) or as a horizontal flow (left-right). Usually, technologies that are hidden from the end-user, like operating systems and hardware, are at the bottom of the stack. While user-facing technologies, such as web platforms, are at the top of the stack.

“Architecture starts when you carefully put two bricks together. There it begins”. Ludwig Mies van der Rohe.

You can even find websites like StackShare where you can share your own tech stack, learn about tech stacks of popular products and how they evolved over time (for example this article shows Slack’s tech stack).

UBER‘sTech Stack shows a typical BIG Data operation

To make life simpler for companies who develop new products, over the years the industry came up with robust and highly integrated generic tech stacks that act as the starting point for common product architectures. For example, 15 years ago when you wanted to build a website, you would start with LAMP stack and then over the years maybe replace or upgrade certain components. These days you might decide to use ELK if you build log management products and Jamstack, MEAN, LAMP or full-stack if you build websites.

ELK Stack

Industrial IoT Tech Stack

Unlike consumer IoT such as Alexa and Fitbit, Industrial IoT (IIOT or Industry 4.0) refers to interconnected sensors in an industrial environment such as manufacturing, utility (oil, gas, water, sewage), transport (automotive, train), construction, agriculture, and even defense. IIOT may also refer to smart cities, smart buildings, and healthcare.

Use case examples:

1. Mobility: Traffic monitoring using a big cluster of IoT sensors (Valerann)
2. Construction: Heavy equipment fleet management (MachineMax)
3. Agriculture: Monitor soil quality (Teralytic)

Types of IoT

Compared to consumer IoT, Industrial IoT is far more challenging:

Industrial environment:

• Deployed in huge sites (area, distances between nodes, number of nodes)
• Remote locations
• Outdoor + Indoor
• Mostly static things, but sometimes assets and machines move.
• Harsh environmental conditions (Temperature, Shock, Vibration, Humidity)
• Retrofitting to old machines, equipment, and infrastructure
• Lack of wired connectivity (Ethernet) and power nodes.
• Difficult to access/install (Safety concerns, Working at height)

Smart Agriculture

Industrial monitoring and control needs:

• Big data: Analysis of thousands of data points
• Security: data theft and equipment/process tampering
• Critical infrastructure: High-reliability
• Realtime processes: Fast response time
• Long product lifetime
• Customized to specific needs / Expandable solution

“It is a capital mistake to theorize before one has data.” ,Sherlock Holmes

Coming up with one, robust, highly integrated IoT tech stack that answers all the needs in the industry is difficult. IoT stacks normally consist of multiple technology layers, each includes various types of technologies:

1. Device layer (Microcontrollers, Sensors, Wireless communication)
2. Data layer (Backend, Database, Machine Learning)
3. Interface layer (Frontend, Fleet Management, Analytics, Visualisation)

“Great things are done by a series of small things brought together.” Vincent Van Gough

There are literally dozens of wireless technologies that are used in IoT solutions. Each provides a different set of range, power consumption, data rate, and cost.

• Long-range: 4G(and 5G), LPWAN (Lorawan, SigFox, NBIOT)
• Low power: BLE, RFID (NFC), LPWAN, Zigbee, Z-Wave, Thread, 6lowpan
• High data rate: 4G, Wifi, Bluetooth

Similarly, there are multiple data layer technologies that are frequently used in IoT solutions:

• Integration and messaging: MQTT, Kafka, RabbitMQ, REST, NodeRed
• Databases: TimeScaleDB, InfluxDB, Postgres, MongoDB, and Redis
• Machine learning: Python + Numpy

And finally, for the interface layer, you can find technologies such as:

• Analytics platforms: Grafana, Kibana
• Visualization tools: D3JS, Vega,
• Frontend stacks: React, Angular
• BI Platforms: PowerBI, Tableau,

Although there can be hundreds of possible combinations of these technologies, there are only a few combinations that work really well together out of the box.

GLAMPP (Grafana, LPWAN, AWS, MQTT, Python, Postgres)

Over the last few years, I worked on several Industrial IoT projects, and somehow, they all converged into the same tech stack:

• Grafana: An open-source analytics and monitoring solution
• LPWAN: Low Powered Wide Area Networks (LoraWan, Sigfox NBIoT)
• MQTT: A lightweight messaging protocol for small sensors
• AWS/Azure: Cloud platform for storage and compute
• Python: A high-level, general-purpose programming language
• Postgres: An open-source relational database

At this point, it’s important to remind us again what we mean by a generic tech stack in the context of this article:

Generic Tech Stack: A robust and highly integrated set of technologies that can act as the starting point for new products.

For example, let’s say we want to develop a new product to monitor / control / manage an industrial process. We can start with these tools and reach a working product (MVP) faster.

GLAMPP Tech Stack

Why LPWAN?

Industrial monitoring requires long-range wireless communication, 100m → 5 Km. This one requirement rules out almost all of the short-medium range of wireless technologies such as Wifi, Bluetooth and Zigbee and leaves out only 4G and LPWAN. Although theoretically mesh technologies such as BLE mesh can reach these ranges, practically it complicates the solution and introduces reliability issues. Another challenging constraint is retrofitting to existing machines/infrastructure and lack of power infrastructure. 4G and 5G use a lot of power and thus are not a viable solution for battery-powered / energy harvesting solutions.

When using LoRa, there are even more advantages compared to 4G:

• It's fairly easy to set up your own private network. Industrial sites are usually located in remote areas where 4G coverage is not always available and additionally face security threats that force them to work in private isolated networks.
• There are many vendors such as Multitech and Kerlink who provide everything needed to put together your own private network with off-the-shelf building blocks in less than 200$.

Why MQTT (Message Queuing Telemetry Transport)?

MQTT has several unique features that make it attractive to IoT applications compared to other protocols such as HTTP or Kafka:

• It’s lightweight: very easy to implement and has a small code footprint.
• It’s fast and realtime!!: Once a connection has been established, the latency for each message is much faster than HTTP
• It implements a bidirectional publish & subscribes messaging. Unlike HTTP where the client can only send messages or poll for new ones, In MQTT, the client can both send and receive messages asynchronously.
• It has a small data overhead (Minimized data packets, low network usage, low power)
• Almost every IoT gateway (and specifically LPWAN) supports MQTT
• AWS IoT Core includes a managed, secured and scalable MQTT broker that allows flexible ruling and integration to other services.
• And finally, it’s very robust

Why Postgres?

It’s quite a surprise that a relational database managed to find it’s way to so many IoT applications, despite the explosion of NoSQL in recent years, and considering the fact that IoT systems normally generate Big Data.

So how come I’ve decided to include Postgres in this stack?

1. It’s SQL!! SQL is the most universal database query language and is particularly effective at data manipulation. Its syntax is intuitive and very close to the English language. It takes between a few hours to a few days for almost anyone to start using it and producing valuable results. When using other languages such as python, data scientists become a bottleneck and the value and insights that you can extract from your data is limited. I can’t express how this one feature is so crucial to the growth of the company. IoT is about data! and access to the data is power! Innovation starts when more people have access to the data and can query it and visualize it. Don’t use databases such as MongoDB and block their access to it.
2. It’s really easy to manipulate time series data: IoT systems almost always generate time-series data. Native Postgres with windows function and with extensions such as TimescaleDB makes it extremely easy to analyze time-series data and to aggregate or compare current values and historical values.
3. It supports spatial data (GIS) and spatial queries. Another important property of any Industrial IoT data is that it’s GIS (location) tagged. When we want to run spatial queries that constrain the result to a certain shape and to run these queries fast, that’s where Postgres with it’s GIS extensions PostGIS excels.
4. It supports partitioning. IoT data is generally categorized as ‘append-only’ time-series data and usually, historical data loses its value fairly fast. Since data that accumulate quite quickly impact the performance of the database, the question of database scalability becomes important. Postgres supports partitioning out of the box now. Since version 10, it’s fairly easy to set up a mechanism that store the data in a daily partition and archive/compress old partitions.
5. AWS RDS supports Postgres natively with features like setting up high availability clusters, launching read replicas, PostGIS support, many security features, monitoring, backups, recovery, etc. Using managed open source databases on top of public clouds like AWS and Azure is a huge benefit because it reduces the risk of vendor lock-in.
6. It’s open-source

“Who has the data has the power” .Tim O’Reilly

Why Python?

IoT is about the data not about the things. For this reason, I’ve decided to include Python as the main language in the stack and not C/C++ which is the common language for embedded (device) programming. Python is one of the most popular languages these days and is used in both web application and data science. It is high-level and simple to learn, portable and easy to deploy and supports both functional programming and object-oriented programming.

• Python is great for data science and includes libraries such as NumPy, Pandas, SciKit-Learn, SciPy, and Tensorflow which are optimized to run machine learning code.
• Python supports interactive, collaborative and online programming with tools such as Jupiter and Google Colab notebooks.
• Python allows data scientists to write production code directly. Avoiding code translation (Matlab → C, R → Java), a process that slows the development process and potentially introduces errors.
• Easy to develop web applications with lightweight platforms such as Flask and more advanced platforms such as Django.
• AWS Lambda natively supports Python as one of the programming languages with unbelievable spin-up times and include common layers to simplify the deployment package.
• Python include libraries such as Psycopg2 that simplifies the connection to Postgres
• Python is open source and has large communities that support it and extend it.

Why Grafana?

Grafana is an open-source, high-performance analytics and interactive visualization software. It’s the easiest way to build an IoT frontend, get user feedback, and interact with the data.

Grafana screenshot

• It supports many data sources among them Postgres, AWS, TimescaleDB, and InfluxDB.
• You can write any SQL query in Grafana and visualize the output either as a graph, a table, a gauge, a single stat, and even display it on a map.
• It’s easy to install, configure and manage
• It includes a built-in configurable alerts engine that can notify groups of users in real-time, either through Slack, email and other methods
• It includes an API that allows you to configure, archive and script a dashboard.
• It’s easy to integrate frames from other websites using AJAX panels
• It provides multiple security features
• It includes manual and automated annotations to log incidents and to collaborate between team members.

Summary

I hope that this short article convinced you that the GLAMPP technology stack provides the best fit for almost every Industrial IoT solution. 5 Years ago, it was almost an impossible (or very expensive) task to build a monitoring solution for these environments, but now that we have LPWAN that sorted the range and power issues; MQTT that delivers the data to the cloud reliably with almost no overhead; Postgres and Python together that provide a simple and powerful data platform, and Grafana that bring all these benefits directly to the end-user, it’s almost no brainer, try and see, it’s free.