Apache Iceberg

The modern world is awash with unimaginable amounts of data created daily by devices, apps and users. Take social media for example: 8.5 billion Google searches, ~100 billion messages on Whatsapp, and over 95 million posts on Instagram. …

“Goals are about the results you want to achieve. Systems are about the processes that lead to results.” — Author: James Clear in his book Atomic Habits With the modern data explosion and the need to derive insights and value from all that data, one thing is clear: a traditional…

In this article, we will look at how the Apache Iceberg table format allows concurrent I/O (writes and reads) operations while providing ACID compliance. We will also go through a simple demo of running concurrent operations and see the results. As data lakes evolved, one of the main constraints they…

We will use Azure Event Grid to implement an event-driven architecture — In our previous article, we demonstrated how an Iceberg table can act as a Kafka topic. We showed that independent Java Writers can produce parquet files in parallel, while a single Bookkeeper attaches these data files to an Iceberg table. The Bookkeeper did this by creating an Iceberg commit. The…

Iceberg Updates Version 1.1 was released, which added API stability guarantees, in addition to these highlights: Puffin Stats Files Branching and Tagging Table Scan Metrics Changelog Scans Spark FunctionCatalog Trino support for the REST catalog, details available here More details can be found here. A spec for “materialized views” in Iceberg was…

This story is a demo of an ongoing PR: https://github.com/apache/iceberg/pull/6449 Apache Iceberg and Delta Lake tables are both popular in big data. The two formats of the table have similarities and differences. Sometimes, users may want to change the format of their table without modifying the data files. …

Use Iceberg API with DuckDB to optimize analytics queries on massive Iceberg tables in your cloud storage — 1. Intro Apache Iceberg is mostly known for making it possible for popular query engines, such as Spark, Dremio and Trino, to reliably query and manipulate records in huge tables stored in data lakes, and to do so in scale while ensuring safe concurrent reads and writes. …

19-Dec-2022 FLiP Stack Weekly Welcome to one of the last newsletters of 2022. It is a honor to receive recognition from my long term blog supersite DZone. Always the best place for #IoT, #BigData, #ML, #Streaming, #ML and other technology deep dives, ref cards and in-depth reports. …

FLiP-Pi-Iceberg-Thermal Apache Iceberg + Apache Pulsar + Thermal Sensor Data from a Raspberry Pi Steps Run Apache Pulsar 2.10.2 (standalone, docker, baremetal cluster, VM cluster, K8 cluster, AWS Marketplace Pulsar, StreamNative Cloud) Run Apache Iceberg (docker, …) 1.1.0 Run Apache Spark 3.2 Deploy Pulsar connector Send data to Pulsar topic Query Iceberg…