Apache Hudi Table Architecture

 

Apache Hudi Table Architecture

Apache Hudi (Hadoop Upserts Deletes and Incrementals) is a data management framework that provides support for incremental data processing and efficient upserts (updates and inserts) and deletes on large datasets stored in data lakes. Hudi’s architecture is designed to provide near real-time data ingestion and supports data versioning and rollback capabilities. Here’s a detailed look at the key components of Hudi’s table architecture:

---

Hudi Table Types

Hudi supports two primary table types:

• Copy-on-Write (CoW): In this type, data is written to new files during updates, and old files are replaced. This is suitable for read-heavy workloads.
• Merge-on-Read (MoR): Here, updates are written to delta files (log files), which are later compacted with base files. This type is suitable for write-heavy workloads where compaction can be deferred.

---

Key Components of Hudi Tables

1. Commit Timeline

• Timeline Management: Hudi maintains a timeline of commits, savepoints, cleanups, compactions, and rollbacks. This timeline helps in tracking the history of changes to the dataset.
• Atomic Writes: Hudi ensures that write operations (insert, update, delete) are atomic. Each write operation creates a new commit on the timeline.

2. Metadata Files

• Commit Metadata: Each commit operation generates metadata files that contain information about the operation, including the files added, updated, or deleted.
• Savepoint Metadata: Savepoints can be created to mark specific points in time that can be rolled back to, providing data versioning and recovery options.

3. Data Files

• Base Files (Parquet Files): In CoW tables, each commit results in new Parquet files that replace old ones. In MoR tables, these files represent the base data before any updates.
• Delta Files (Log Files): In MoR tables, updates and deletes are written to delta files, which are later compacted with the base files to create new base files.

4. Indexing

• Bloom Filter Index: Hudi uses bloom filters to index records in the data files, enabling efficient record lookups during updates and deletes.
• External Index: Hudi can also integrate with external indexing systems like Apache Hive to manage large datasets efficiently.

5. Compaction

• Scheduled Compaction: In MoR tables, periodic compaction operations are performed to merge delta files with base files, improving read performance by reducing the number of files to be scanned.
• Inline Compaction: Compaction can also be performed inline with write operations to ensure that base files are always up-to-date.

---

How Hudi Works

1. Data Ingestion

   Data is ingested into Hudi tables using ingestion jobs. These jobs can handle inserts, updates, and deletes.

   • For CoW tables, each ingestion job writes new Parquet files.
   • For MoR tables, ingestion jobs write delta files that record changes.

2. Indexing

   Hudi uses bloom filters and other indexing techniques to enable fast lookups of records, making updates and deletes efficient.

3. Compaction (MoR Tables)

   Delta files created during updates are periodically compacted with base files to create new base files.

   • Compaction operations can be scheduled or performed inline to balance write performance with read efficiency.

4. Commit Timeline

   Each data operation creates a new commit on the timeline, ensuring atomic and consistent updates.

   • The timeline helps in tracking the history of changes and supports operations like rollback and savepoints.

5. Querying

   Hudi tables can be queried using standard SQL queries through integration with query engines like Apache Hive, Presto, and Apache Spark.

   • The timeline and indexing ensure that queries are efficient and reflect the latest state of the data.

---

Key Benefits

• Efficient Upserts and Deletes: Hudi supports efficient handling of updates and deletes, which is crucial for data lake management.
• Incremental Data Processing: Hudi enables incremental data processing, allowing only new or changed data to be processed, improving performance.
• Data Versioning and Rollback: Hudi provides versioning and the ability to rollback to previous states, ensuring data consistency and recovery options.
• Scalability: Hudi is designed to handle large-scale datasets efficiently, making it suitable for big data applications.

---

Conclusion

Apache Hudi’s architecture, with its commit timeline, metadata management, efficient indexing, and support for both CoW and MoR tables, provides a robust framework for managing large-scale data lakes. It enables efficient upserts and deletes, incremental data processing, and ensures data consistency and reliability, making it an ideal choice for modern data management needs.

---