Posts

Showing posts with the label Iceberg Architecture: A Three-Tier Structure

Iceberg Architecture: A Three-Tier Structure

  Iceberg Architecture: A Three-Tier Structure Apache Iceberg is an open table format designed for large-scale analytic datasets. It provides a reliable and efficient way to manage big data in data lakes. Iceberg tables have a three-tier structure comprising metadata files, manifest list files, and manifest files. Here’s a detailed explanation of each tier and how they work together: 1. Metadata Files Metadata files are at the top level of Iceberg's architecture. They store information about the table’s schema, partitioning, and snapshot references. Table Metadata : This includes the table schema, partitioning information, and properties. It keeps track of the current snapshot ID and a list of all snapshots. Snapshots : Each snapshot points to a specific state of the table at a given time. Snapshots reference manifest list files and contain information about the data files added, deleted, or modified in that snapshot. Versioned : Iceberg supports versioned metadata files, which me...