Building a modern geospatial IoT platform at scale using open-source software, open standards, and open data

Last week, I had the opportunity to attend the OSGeo Alberta Chapter inaugural meeting and talk about how we, at SensorUp, are building a modern and scalable geospatial IoT platform using an open ecosystem. OSGeo Alberta Chapter is a community-driven initiative to support local geospatial users and developers and promote the adoption of open-source geospatial software and technology in Alberta and surrounding areas.

What is an Open Ecosystem?

SensorUp has adopted an open ecosystem, which has three main components: open-source software, open standards, and open data. Before I get into more detail, I should note that there are some similarities and differences between the terms “free” and “open” — people sometimes use them interchangeably, however, “free” does not always convey the value of an “open” ecosystem. For example, Google Docs is free software, and many people around the world use it, but it is not open source. Or you may get some geospatial data in (ESRI’s) geodatabase format free of charge, but that does not mean geodatabase is an open standard. Another example is transit updates that the City of Calgary was sending out in PDFs back in 2013 when the flooding happened; it was all free but not open data.

Open is all about freedom — freedom to use, freedom to modify, and freedom to share the modifications. It is similar to how science works; when you want to start a research project, you do a literature review to evaluate the state-of-the-art and find your starting point. You likely find some related work that others have done. You use their results and try to find ways to improve them. Then you publish your discoveries and share your scientific results with others. And this cycle goes on and on.

Consider Leaflet as an example — an open-source web mapping library developed by Vladimir Agafonkin. It is a small library that has just enough functionality to work efficiently on different web browsers and mobile devices. It supports a plugin-based development approach that allows developers to extend its functionality by writing plugins. And they do. Today, there are a lot of open-source plugins available for Leaflet. Leaflet is an open-source library that gives developers the freedom to take as much as they want, and if they want more, they can build on top of it and share it with others.

Open is all about community — a group of like-minded people who share a common goal. A great example is OpenStreetMap, which is the encyclopedia of maps, created and maintained by a community of volunteers worldwide. It started small in the UK but proliferated. One of the main reasons behind OpenStreetMap’s popularity is because it is an open data platform that allows people to use, modify, and share the modifications. And with that, people feel empowered because they are in control.

There are some areas in the world that OpenStreetMap has much better coverage than proprietary geospatial data providers like TomTom/Google. A well-known example is OpenStreetMap’s better coverage for the Olympic sites in Sochi in 2014, compared to Google Maps, which gained so much attention in the media at the time.

Nowadays, OpenStreetMap has more than 5 million registered users who contribute to this open data platform every day.

Open is all about inclusion. It does not mean that you have to abandon your existing, proprietary database (e.g., Oracle) to work with an open-source GIS (e.g., QGIS) or replace your current enterprise map server (e.g., ArcGIS for Server) with an open-source solution (e.g., GeoServer) to able to work with an open-source mapping library (e.g., Leaflet or Mapbox GL JS). It allows you to go hybrid. In an open ecosystem, you can choose whatever fits your needs best.

There are some benefits when using an open ecosystem. Open-source software provides cost optimization, which is critical for small-size companies, especially startups. You do not have to worry about licensing or the number of users, so you can scale out easily. You get community support — it could be answering questions on StackOverflow or even contributing to your project if it is open source.

Open standards provide flexibility in terms of system components you can use in your architecture. As long as they are standard-compliant, they can be easily integrated with existing systems or even get replaced. For example, as long as your map server publishes data using WMS, from an integration standpoint, it does not matter if you have an open-source or proprietary solution (GeoServer vs. ArcGIS for Server) because they use a standard protocol to publish data. If all sensor providers supported SensorThings, then sensor data integration would be much easier and painless. Additionally, with flexibility comes reusability where you can use your components in other projects.

Finally, open data provides cost optimization and low licensing fees. Last year it was reported that Wikipedia’s annual Google Maps API bill was around $250K while the annual cost of running OpenStreetMap was less than $120K at the time. Plus, there are contributions from the community that help improve data quality in the long run.

SensorUp Solution

At SensorUp, we use open source software, open standards, and open data to build a geospatial IoT platform for the 21st century. The SensorUp Solution is an end-to-end software platform for data ingestion, processing, and visualization. It connects to various data sources, from sensors to APIs. It takes care of the ETL process, from data extraction to transformation and load. It unifies various datasets (observations and data streams) into a standard data format, ready for aggregations, enrichment, and analytics. Finally, it provides a dashboard environment in that users can create data visualizations, reports, and workflows. And all of this is done using an open ecosystem.

From an architectural perspective, the platform has five main components. SensorUp Ingress is the entry point of the system. It manages the integration between the SensorUp solution and different data sources, including sensors, databases, FTPs, and APIs. It extracts data from all those data sources and transforms it into the OGC SensorThings data format, and loads it into SensorUp Unified Pipeline.

SensorUp Unified Pipeline enables various real-time, event-driven processes as well as historical archival and retrieval processes. At its core, it is a stream processing pipeline that analyzes and aggregates data in real time and provides an API endpoint for downstream applications to consume various data using newline-delimited GeoJSON.

SensorUp Rules Engine is a workflow management engine that enables notifications and actions. It provides an interface to create chains of conditional statements by defining evaluations (e.g., setting thresholds) and actions (e.g., sending emails).

SensorUp API provides an open and unified way to access data, whether it is historical or real-time observations. It also provides an interface to interact with Rules Engine. SensorUp API also manages identity access management (authentication and authorization).

Finally, SensorUp Explorer provides an easy-to-use and intuitive environment for users and decision-makers to visualize various data streams using different types of maps and charts and receive alerts and notifications on a dashboard. Mapbox GL JS, deck.gl, and turf.js are some prominent open-source geospatial libraries used in SensorUp Explorer.

SensorUp’s engineering team uses a cloud-native architecture to design, configure, and deploy the system components. AWS is SensorUp’s cloud provider of choice.

Wrap Up

From my perspective, you get three main advantages when adopting an open ecosystem. You get the freedom to choose different software components, datasets, or even standards based on your needs. It gives you flexibility. You get a great community of volunteers who do their best to help you out with answering technical questions, contributing to the codebase, or the open data platform you might have. And you get inclusion, where you can use different software components and datasets, from open-source to proprietary, in a software system. And that prevents vendor lock-in. You can have a solution that scales well without having to worry about software cost and licensing fees.

There are also some challenges when it comes to using open-source software, open data, and open standards.

The first challenge relates to resistance to change. People usually like to use what they know best, and that is likely what they started using years ago, which is normal. ESRI products are an example in the geospatial industry — no matter how great OSGeo products are, it is human nature for people to be reluctant to adopt new software and technology. And the bigger an organization is, the harder it gets to encourage people to explore new things. So if you want to start exploring, start small, with a single application, and give it a shot.

Second, open-source tools can have a steep learning curve, and it may take a while for open standards to be established. So give yourself or your developers the space to learn and experiment and try different approaches.

The final challenge focuses on marketing. Open-source has no/little marketing department. People usually use dominant, enterprise products, especially in corporations, and adopt some jargon related to those proprietary solutions when talking about functionalities, capabilities, or objectives. For example, instead of saying, “you need a web map server capable of handing vector data,” they say, “you need a GIS, ArcGIS Server, perhaps.” Or instead of saying, “you need a spatial database that supports versioning,” they say, “you need a SQL database and SDE.” However, from a system architecture viewpoint, you end up with better designs if you model your system based on functionality, and then figure out what product mix fulfills your needs.