PinnedBurak DoguBuilding a Data Warehouse Pipeline with Kafka, Cassandra, Airflow and SnowflakeIntroduction4 min read·Oct 24, 2023----
Burak DoguAWS Real-time views data processing with Kinesis TechsIn this article, I will take you through a project where we leveraged AWS Kinesis techs especially with Apache Flink, Lamda, S3 and…4 min read·Feb 17, 2024----
Burak DoguReal-time processing using by AWS Kineses Data Streams, Firehose and Lambda to store S3 and…In the world of big data, effieciently managing and processing data streams is paramount. AWS offers a plethora of services that can be…3 min read·Feb 11, 2024----
Burak DoguData processing using by Cloud Functions with different trigger methodIn this article, we will explore a comprehensive guide on how to effectively use Google Cloud Platform(GCP) for a specific scenario…5 min read·Feb 7, 2024----
Burak DoguBatch Retail Data Process with DataflowIn this article, assumed a retail data is used for batch apache beam process with window progress. Pipeline have been created to perform…3 min read·Jan 13, 2024----
Burak DoguHow to avoid duplicates in real-time process with Kafka and SparkIntroduction3 min read·Dec 6, 2023--1--1
Burak DoguAirflow Incremental Dataset and test with PytestAirflow is a powerful and versatile platform that plays a crucial role in modern data engineering and workflow automation. In the context…2 min read·Oct 24, 2023----
Burak DoguLog files transferring to Postgresql using by Kafka,Elasticsearch,Filebeat,Spark and orchestrating…In today’s rapidly evolving technological landscape, managing and analyzing vast amounts of data has become an essential aspect of…2 min read·Oct 24, 2023--1--1
Burak DoguReal time Kafka data to Cassandra using by Spark and AirflowIntroduction We generate fake person data with python packages faker and send to real time kafka. We also schedule and orchestrate this…2 min read·Oct 5, 2023----