Published in


Is this your next data logistics platform architecture?

Photo by Jacques Dillies on Unsplash

The basic 4 — tier architecture

Creating knowledge from data has at least three steps, first ingest data into the platform, second integrate various data sources, and third present a unified consumable and understandable data model to your consumer systems.

Basic schema of a data logistics platform architecture

Tools to implement this data logistics platform on Google Cloud

First, which tools are relevant to implement a data integration platform as described above. Google has the BigQuery service, which can be your go-to place to implement all 4 layers, as long as you only have data in sources that are available as connectors to Google BigQuery and as long as you what to implement single direction data transportation.

Example implementation on Google Cloud Platform

Tools to implement this data logistics platform on Microsoft Azure

Not a google user? Well, let’s see how to implement a data logistics platform on Microsoft Azure! First, the data ingestion part, similar to GCP there is Azure Functions to ingest data from complex APIs or data lakes. Speaking of data lakes, together with Azure Storage this is the go-to solution to dump data from sources outside of Azure. In case of simple APIs and already available connectors, the Azure Data Factory can be a part of the data ingestion layer.

Example implementation on Azure


The 4-tier architecture and its implementation on a hyperscaler comes always with the need for DevOps or CI/ CD processes. Therefore the infrastructure needs to be implemented via an IaC solution. Depending on the selected solution this comes at the cost of a more or less complex implementation of CI/ CD workflows and also at cost of writing tests to verify the state of the infrastructure, the configuration as well as the data.



Everything connected with Tech & Code. Follow to join our 1M+ monthly readers

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Otrek Wilke

Data Engineering made easy. Writing about things learned in data engineering, data analytics, and agile product development.