PinnedBurak DoguBuilding a Data Warehouse Pipeline with Kafka, Cassandra, Airflow and SnowflakeIntroductionOct 24, 2023Oct 24, 2023
Burak DoguAWS Real-time views data processing with Kinesis TechsIn this article, I will take you through a project where we leveraged AWS Kinesis techs especially with Apache Flink, Lamda, S3 and…Feb 17Feb 17
Burak DoguReal-time processing using by AWS Kineses Data Streams, Firehose and Lambda to store S3 and…In the world of big data, effieciently managing and processing data streams is paramount. AWS offers a plethora of services that can be…Feb 11Feb 11
Burak DoguData processing using by Cloud Functions with different trigger methodIn this article, we will explore a comprehensive guide on how to effectively use Google Cloud Platform(GCP) for a specific scenario…Feb 7Feb 7
Burak DoguBatch Retail Data Process with DataflowIn this article, assumed a retail data is used for batch apache beam process with window progress. Pipeline have been created to perform…Jan 13Jan 13
Burak DoguHow to avoid duplicates in real-time process with Kafka and SparkIntroductionDec 6, 20231Dec 6, 20231
Burak DoguAirflow Incremental Dataset and test with PytestAirflow is a powerful and versatile platform that plays a crucial role in modern data engineering and workflow automation. In the context…Oct 24, 2023Oct 24, 2023
Burak DoguLog files transferring to Postgresql using by Kafka,Elasticsearch,Filebeat,Spark and orchestrating…In today’s rapidly evolving technological landscape, managing and analyzing vast amounts of data has become an essential aspect of…Oct 24, 20231Oct 24, 20231
Burak DoguReal time Kafka data to Cassandra using by Spark and AirflowIntroduction We generate fake person data with python packages faker and send to real time kafka. We also schedule and orchestrate this…Oct 5, 2023Oct 5, 2023