Data Lake

In the world of Big Data and data science, the need for efficient, high-performance data processing frameworks is more crucial than ever. In this domain, Apache Arrow has emerged as a revolutionary open-source project. This blog post will explore the core principles of Apache Arrow, its design and functionality, and…

Nowadays, advanced analytics platforms are expected to cope with traditional and modern workloads (streaming included), while jointly handling structured, semi-structured and unstructured data on top of scalable and affordable storage. These architectures guarantee correctness while delivering high performance in a wide range of scenarios, avoiding the overhead of implementing multiple…

What is a data lakehouse? The history of Data Management! In this article, we will learn about 1. The Origin and purpose of Data Lakehouse 2. The Challenges of managing Big Data The History of Data Management And Analytics To better understand what data lakehouse is, we need to explore the history of Data management and analytics.

Introduction In today’s data-driven world, organizations face the challenge of managing the ever-growing volume of data. Efficient data storage solutions are crucial for making informed decisions based on this data. …

A Data Mesh is a novel, decentralized approach to data architecture that treats data as a product and distributes data ownership across an organization. Unlike centralized data systems like a data lake, a data mesh allows each team to manage and be responsible for their own data, promoting better data…

Getting Up to Speed on Real-Time Machine Learning with Spark and SBERT Learn more about real-time machine learning by using this approach that uses Apache Spark and SBERT. 8 Data Lake Vendors to Make Your Data Life Easier in 2023 To make your data management processes easier, here’s a primer…

Today it can seem like a buzzword onslaught in the data space with terms like Data Mesh, Data Lakehouse, and many more being thrown out with every vendor trying to make their claim that they provide “the answer.” However, in this article, I will discuss the underlying reality that does…

Iceberg updates Apache Iceberg 1.3 was released. Spark now supports the Iceberg UUID type as a string type, ensuring compatibility with tables created using Trino. Some additional noteworthy updates are mentioned below: Flink version 1.17 support was added Spark version 3.4 support was added Added support for Spark’s new TIMESTAMP_NTZ type using…

Editor’s note: Kai Waehner is a speaker for ODSC Europe this June. Be sure to check out his talk, “Apache Kafka for Real-Time Machine Learning Without a Data Lake,” there! The combination of data streaming and machine learning (ML) enables you to build one scalable, reliable, but also simple infrastructure…