Member-only story
Build a real-time data analytics pipeline with Airbyte, Kafka, and Pinot
Learn how to use Airbyte, Kafka, and Pinot to build a data pipeline for a user-facing analytics dashboard.
This post was originally published at Airbyte.
Typically, a real-time analytics pipeline consists of several components, including ELT pipelines, an event streaming platform, and an analytical database that can answer queries at scale. However, building and maintaining such a platform is expensive and demands significant engineering effort and time.
Having open-source data engineering and analytics tools at your disposal helps reduce the cost of running a real-time analytics pipeline. Having access to the source code avoids vendor-locking, giving you the flexibility to customize the solution to tailor your organizational needs. This article explores three popular open-source products in the data space, Airbyte, Apache Kafka, and Apache Pinot, to build a user-facing e-commerce dashboard that updates in real-time.
Airbyte is an open-source data integration platform capable of moving data from OLTP databases such as MySQL to destinations such as Apache Kafka using change data capture (CDC) with low latency. Apache Pinot is an open-source OLAP database capable of ingesting…
