Hudi

Apache Hudi is the open-source data lakehouse platform created at Uber for efficient upserts and incremental processing on large datasets stored in object storage — solving the specific challenge of applying real-time database changes (inserts, updates, deletes) to massive Parquet-based data lakes without rewriting entire partitions on every change.

What Is Apache Hudi?

• Definition: A data lake storage framework that provides efficient upsert (update + insert) and delete operations on large datasets stored in HDFS or object storage — using a record-level index to locate which file contains a specific record and updating only that file rather than rewriting entire partitions.
• Origin: Created at Uber in 2016 to solve the "How do we apply driver payment updates and trip corrections to our 100TB+ data lake in near real-time?" problem — donated to Apache in 2019.
• Record Index: Hudi maintains a record-level index (HBase or in-file) mapping each record key to its physical file location — enabling point updates to individual records without full partition rewrites.
• Table Types: Hudi offers two table types optimized for different access patterns: Copy-on-Write (COW) for read-heavy workloads and Merge-on-Read (MOR) for write-heavy streaming use cases.
• Incremental Queries: Consumers can query "What records changed in the last 15 minutes?" rather than reprocessing the entire table — critical for streaming ETL pipelines and real-time ML feature updates.

Why Hudi Matters for AI/ML

• Real-Time Feature Updates: Update individual user features (latest purchase, recent click, current balance) in the feature store within minutes of the triggering event — Hudi's upsert handles the "update this one record" operation efficiently.
• Streaming Ingestion: Kafka → Spark Structured Streaming → Hudi table pipeline: continuously ingests CDC events from databases into a queryable analytical table updated in near-real-time.
• Incremental Training: ML pipelines can consume only new/changed records from Hudi tables since the last training run — avoiding reprocessing terabytes of historical data to incorporate daily updates.
• GDPR Compliance: Delete a specific user's records across all Hudi tables without partition rewrites — Hudi's delete operation marks records as deleted in the index and filters them from queries.
• Time Travel: Audit training data state at any past point — Hudi maintains timeline metadata enabling point-in-time queries for debugging model drift.

Core Hudi Concepts

Table Types:

Copy-on-Write (COW):
- Writes rewrite affected Parquet files with updates applied
- Read-optimized: readers always see clean Parquet files
- Write amplification: expensive for high-frequency updates
- Best for: analytics workloads with infrequent updates

Merge-on-Read (MOR):
- Writes append delta log files (Avro format) rather than rewriting Parquet
- Reads merge base Parquet with delta logs on the fly
- Write-optimized: extremely fast ingestion for streaming
- Best for: streaming CDC ingestion, near-real-time use cases

Hudi Timeline (Transaction Log):
- Ordered sequence of actions: commit, compaction, clean, rollback
- Every committed instant is immutable with timestamp, action type, and state
- Incremental queries specify a start instant to get only subsequent changes

Incremental Query Pattern:
hudi_df = spark.read.format("hudi")
.option("hoodie.datasource.query.type", "incremental")
.option("hoodie.datasource.read.begin.instanttime", "20240101000000")
.load("/path/to/hudi/table")

Compaction:
- MOR tables periodically compact delta logs back into Parquet base files
- Scheduled as async background job to avoid blocking ingestion
- Reduces read-time merge overhead as delta logs accumulate

Hudi vs Alternatives

Apache Hudi is the streaming-first data lakehouse platform that makes real-time upserts on massive datasets practical — by maintaining a record-level index and providing both copy-on-write and merge-on-read table types, Hudi enables ML teams to build near-real-time feature stores and continuously updated training datasets on top of object storage without the prohibitive cost of full-partition rewrites.