Understanding the IoT Journey to design your serverless architecture
Executive Summary
- Currently IoT projects are stuck by the cost and complexity of the delivery of an IoT platform.
- Opting for a cheap and non scalable solution drives the technical debt to the roof as your project becomes successful.
- Serverless architecture can be a great solution to build a flexible, reliable and cost-efficient platform with minimal investment in infrastructure and software while greatly improving time to market.
- Designing an IoT platform using a serverless architecture requires a good understanding of the IoT journey which is described in this post
Introduction
Over the past 8 years I have been supporting customers on all the aspects of their Internet of Things (IoT) journey — from identifying and estimating the business value of IoT to deploying and running large scale IoT projects.
Taking my IoT experience in the light of the serverless trend, I am struck by the new possibilities that serverless brings to the IoT digitalization that we are facing today. Through this paper, I will summarize the needs during an IoT journey and explain the benefits of serverless infrastructure.
Needs of an IoT journey
An IoT journey is a succession of project milestones with different requirements. I will first elaborate on those milestones and then shed the light on the market reality of IoT to then summarize on the technical needs for an IoT project.
Project milestones
All successful IoT journeys I experienced went through the 3 main stages below:
At each step you have different goals and do different technical requirements.
Besides the different technical requirements of each milestone, there are different levels of investment. An IoT journey that is built on value creation, there is an important failure risk associated to it and as a consequence, during the early stages of the journey the investment is low and so you accumulate a technical debt that becomes very complicated to solve at scale as described below.
Now I have described the challenges of an IoT project, let’s take a look at the market reality of the IoT projects
Market reality
The first reality of IoT projects is that 60% of the perceived value resides on the industry specific application and the Business Intelligence (BI). All the underlying wiring has very little perceived value for the customers. Below is a high level architecture of an IoT platform with the perceived value for each blocks:
First conclusion, customers do not want to pay for the underlying wiring.
The second reality of IoT projects is that most use cases are very specific and require custom development. By opposition to smart phones, that has a couple of hardware variants, IoT will require a large variety of different devices, just like we’ve seen it for Machine to Machine (M2M). And this phenomenon is amplified by the fact that the market is still very immature!
Second conclusion, the underlying wiring must handle a lot of different networks and communication protocols.
The third reality of IoT projects is that most of them start with the ROI driver and eventually ends up with the value creation driver. As a consequence, costs must be very much contained.
Third conclusion, the underlying wiring needs to become a cheap commodity.
The last reality is that IoT is HARD! There are so many components that can go wrong in an IoT journey: hardware, firmware, platform, connectivity… As a consequence, the IoT platform must be easy to develop.
Last conclusion, the underlying wiring needs to be simple to implement.
Technical requirements
As explained in the two previous chapters, there is a complex challenge to solve: on one hand IoT customers need a complex and scalable IoT underlying infrastructure that is hardly duplicable and on the other they don’t want to invest in it at the beginning of their journey.
To summarize you need a system that is:
+ Billed on a pay per use
+ Easily scalable to handle traffic peak even at low volume
+ Future proof to manage different devices and different networks
+ Real time, event driven and high availability
+ Easy and simple to deploy and maintain
Historically, building such a system was very costly!
The serverless revolution
What is this serverless concept?
The concept of serverless is that you just have to upload a bunch of code that reacts to event driven solicitation and the cloud provider manages everything else for you. The serverless was initially mainstreamed by AWS Lambda functions and then all major cloud providers offered a similar service. Below is a simple diagram to explain you the recent evolution of cloud providers towards serverless Function as a Service (FaaS):
As you can see it allows you to focus on the strict minimum necessary: your code!
Why is it a perfect fit for IoT?
Let’s consider the technical requirements of an IoT platform as explained earlier:
- Billed on a pay per use → FaaS are billed by the seconds
- Easily scalable to handle traffic peak even at low volume → The cloud provider takes care of everything
- Future proof to manage different devices and different networks → You can easily create a new function and connect it to messaging bus
- Real time, event driven and high availability → FaaS are designed to do just that
- Easy and simple to deploy and maintain → FaaS allows you to just push your code and not bother about anything else
As a consequence, you can easily start your IoT journey without burning too much cash and without accumulating technical debt.
How to use it?
Depending on Cloud providers there are different solutions available but the main 3 (AWS, GCP and Azure) all have the components to drive an architecture as described in this chapter.
The high level architecture is basically the following:
If we focus on the functions side, the data flow will be the following :
In order to design a flexible architecture that is highly scalable, the best architecture design would look like this:
Such an architecture allows you to have a very flexible service oriented infrastructure that will be able to scale over time and meet the evolving needs of an IoT journey.
Next steps
In this architecture I didn’t focus on the device management part nor on the app part of the IoT platform. For those 2 parts, there are several vendors providing solutions and I will focus on it in a later post.
You will find code examples to build this architecture at the following github repos:
Please comment and give your feedback!