This interactive tutorial provides a comprehensive deep dive into Apache Iceberg, an open table format central to building decomposed database architectures. The presenter contrasts Iceberg with traditional Hadoop and Hive approaches, highlighting its ability to separate compute from storage and foster true interoperability across diverse query engines. The core of the tutorial dissects Iceberg's metadata-driven architecture, illustrating how it enables advanced features like schema evolution without data rewrites, efficient update and delete operations via "delete files," historical time travel to any table state, and hidden partitioning for optimized query performance. Practical demonstrations include analyzing UK housing price data against other asset classes and a detailed walkthrough of integrating Iceberg with Kafka for stream processing, offering a unified platform for both real-time events and historical analytics. The session emphasizes how Iceberg allows data practitioners to use their preferred tools while leveraging a powerful, flexible data foundation.
Exploring Apache Iceberg: A Deep Dive into Open Table Formats
0:00 Introduction to Iceberg: The presenter introduces an interactive tutorial on Apache Iceberg, highlighting its role in enabling a decomposed database architecture by separating compute and storage, and stressing the flexibility it offers through open standards.
0:42 The Evolution to Decomposed Databases: The discussion traces the origins of Iceberg from the limitations of Hadoop and Hive, explaining how Iceberg addresses the inflexibility of tightly coupled compute and storage by standardizing data access.
5:01 Motivating Example: UK Housing Prices: The tutorial uses a real-world dataset of UK housing prices from gov.uk to demonstrate Iceberg's capabilities.
6:38 Iceberg Architecture in a Snapshot: A detailed walkthrough of Iceberg's metadata-driven "onion" architecture, which underpins its advanced features.
6:59 Iceberg Catalog: The top layer responsible for tracking tables and their current versions, with various supported implementations (e.g., AWS Glue, Snowflake, Hive metastore, LakeKeeper).
8:02 Schema Definition and Namespaces: Iceberg uses namespaces and tables (instead of schemas) and defines data schemas with nested fields, field IDs, data types, and optional documentation strings.
10:19 Data Writing Operations: Iceberg supports append, overwrite, and upsert methods, enforcing strict schema validation and leveraging the catalog to manage temporary credentials for object storage (e.g., MinIO, S3).
12:18 The Metadata Layers: The core "magic" involves metadata.json (snapshots), manifest lists, and manifests. Snapshots store table-level information and point to manifest lists; manifest lists detail manifest files; and manifests contain rich metadata about individual data files (e.g., row counts, min/max values), enabling query pruning without reading full files.
17:58 Data Immutability: Iceberg maintains data immutability by writing new data files and updating metadata, rather than modifying existing files directly, making operations cheaper and ensuring data integrity.
20:03 Interoperability with Query Engines: A key benefit of Iceberg is its broad support across various query engines and programming languages, including Python's pyiceberg, Polars, DuckDB, SQL engines like Trino, and distributed frameworks like Daft, allowing users to choose the best tool for their needs.
28:51 Schema Evolution: Iceberg enables seamless schema changes (e.g., adding or renaming columns) by only updating metadata, without requiring costly data rewrites.
32:57 Efficient Updates and Deletes: Modifications are handled using "delete files" (equality or positional deletes) rather than direct file alterations, preserving the immutability of existing data and optimizing performance.
35:10 Time Travel & Rollbacks: Iceberg's historical metadata allows users to query data "as of" any previous snapshot ID or timestamp, facilitating debugging, auditing, and easy rollbacks to past table states.
38:05 Branches and Tags: The format supports named references to specific snapshots (similar to Git branches and tags for data), providing human-readable aliases for important data versions and enabling complex workflows like data branching and merging.
40:15 Table Maintenance and Optimization: Iceberg includes built-in functions for cleaning up (expiring old snapshots, removing orphan files) and optimizing data layout (merging small files into larger, more efficient ones) to combat the "small files problem" common in object storage.
42:55 Hidden Partitioning: Iceberg introduces "hidden partitioning" where data transformations (e.g., by month, year, or hash) are registered as metadata. The query engine automatically uses this information for efficient data pruning without requiring users to specify partitions in their queries, improving performance.
48:04 Evolving Partitioning: Iceberg allows for changes in partitioning strategy over time (e.g., from monthly to yearly partitions) without forcing a full data rewrite, seamlessly handling mixed partitioning schemes.
51:57 Practical Housing Price Analysis: The tutorial demonstrates a full data pipeline, using Trino for data engineering tasks (ETL, creating fact/dimension tables) and Polars for detailed analysis, such as calculating house profits and comparing them to other asset classes like gold.
1:05:09 Streaming Data Integration with Kafka: Iceberg is presented as a strong solution for integrating streaming data from Kafka, enabling real-time analytics by batching messages into Iceberg tables and providing historical context for long-term analysis.
1:10:50 Challenge and Prize: Participants are challenged to extend the housing price analysis by comparing profits with other asset classes, with Danish cookies as a prize for the most interesting use case.