How we manage AirQo microservices

A subset of AirQo’s architecture shows the information flow between microservices.
A subset of AirQo’s architecture showing information flow between microservices.
  1. Service Monitoring and performance: Service monitoring and performance tracing in microservices data pipelines can be complicated especially in such a case where we have multiple decoupled applications and data pipelines with data dependencies on each other. How would you know that a core microservice has malfunctioned ?, How would you tell that a particular stage of the pipeline is broken?. You can only improve on something you have prior information about.
  2. Data inconsistency: The one-to-many relationship between the data and microservices creates room for data inconsistency across the microservices, causing information mismatch on our digital platforms and access points.

What tools do we use to solve challenges arising from the microservices architecture?

A. For Streaming and Workflow management

We rely on two open-source tools Airflow and Kafka for streaming and pipeline management.

  1. Kafka
    We use Kafka, an open-source distributed event store and stream-processing platform to handle real-time air quality data sourced from the AirQo sensors and partner organizations. It enables the streaming of huge amounts of data with the advantage of intermediate data storage between services. In case a microservice is not available at any particular time and some data is not processed, the unprocessed data can be stored on Kafka topics and processed as soon as the service gets online. This ensures we don’t lose out on any data due to service unavailability.

B. CI/CD and Service monitoring

We leverage Kubernetes and GitHub actions to ensure continuous deployment of applications as soon as they are ready. This helps to reduce the time we spend on deploying applications.

C. Data Storage

Our data storage units have been set up and configured to support handling vast amounts of data while putting into consideration cost and performance issues. We use BigQuery as our data warehouse with tables clustered and partitioned. The microservices store data using a database-per-service approach and rely on PostgreSQL and Mongo depending on the use case.

Explore our digital tools. Learn about the quality of air around you. Click here to get started.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
AirQo Engineering

AirQo Engineering

Engineering and technology articles for developers, written and curated by AirQo engineers