Data Lake

Nowadays, advanced analytics platforms are expected to cope with traditional and modern workloads (streaming included), while jointly handling structured, semi-structured and unstructured data on top of scalable and affordable storage. These architectures guarantee correctness while delivering high performance in a wide range of scenarios, avoiding the overhead of implementing multiple…

Getting Up to Speed on Real-Time Machine Learning with Spark and SBERT Learn more about real-time machine learning by using this approach that uses Apache Spark and SBERT. 8 Data Lake Vendors to Make Your Data Life Easier in 2023 To make your data management processes easier, here’s a primer…

Today it can seem like a buzzword onslaught in the data space with terms like Data Mesh, Data Lakehouse, and many more being thrown out with every vendor trying to make their claim that they provide “the answer.” However, in this article, I will discuss the underlying reality that does…

Editor’s note: Kai Waehner is a speaker for ODSC Europe this June. Be sure to check out his talk, “Apache Kafka for Real-Time Machine Learning Without a Data Lake,” there! The combination of data streaming and machine learning (ML) enables you to build one scalable, reliable, but also simple infrastructure…

Iceberg updates Apache Iceberg 1.3 was released. Spark now supports the Iceberg UUID type as a string type, ensuring compatibility with tables created using Trino. Some additional noteworthy updates are mentioned below: Flink version 1.17 support was added Spark version 3.4 support was added Added support for Spark’s new TIMESTAMP_NTZ type using…

Two potential data lake architecture In today’s data-driven world, building efficient and scalable data lakes is crucial for organizations looking to unlock the full potential of their data. With the advent of new technologies and innovations, Oracle Autonomous Data Warehouse (ADW) has emerged as a powerful solution for creating robust data lakes. …

Disclaimer: The project is for personal interest only and not for deployment Intention In the era of data, a data lake is always the backbone of any company. Many companies posses a mature data platform which typically includes data lake, orchestrator and the underlying compute engine. If you want to step…

We need to establish a few things to understand why the open-source data catalog, Project Nessie, matters so much. Why do Data Lakehouses Matter? The amount of data and use cases for that data are exploding faster than any single tool can develop sufficient support for. However, many tools support different use cases like BI…

Introduction It is a known fact that write to read ratio in an analytical system is generally 1:100 or even more and the reads are done by various consumers for different purposes ranging from real time dashboarding to weekly reports to downstream ETL pipelines and much more. So, to cater to…