Apache Iceberg

Databricks implemented the data Lakehouse architecture through its open-source project Delta Lake.

Lakebase is a new type of database form proposed within the Databricks ecosystem, whose core concept is to build Postgres-compatible transactional database capabilities on top of a data lakehouse architecture.

Apache Iceberg and Apache Hudi, together with Delta Lake, are collectively known as the three major open formats of the lakehouse era.

Delta Lake uses a transaction log (_delta_log), Apache Iceberg uses manifest files and a metadata tree, and Apache Hudi uses a timeline (Timeline) as the mechanism for implementing the metadata layer.

Migrating online business from InnoDB to Iceberg is not just a matter of changing the connection string, but requires redefining the trade-offs among consistency, latency and cost.

Iceberg provides index formats such as Puffin to supplement Bloom filters or finer-grained statistics, in order to improve query pruning effectiveness.

Iceberg is an open table format for data lakes, and its metadata hierarchy is: table metadata JSON → manifest list → manifest file → data file, where each manifest file records data file paths, partitions, column-level statistics and file-level statistics.

Iceberg's manifest files are Avro files, and each entry contains the file path, the partition it belongs to, record count, file size, and statistics such as the upper and lower bounds and null counts for each column; query planning uses these statistics and partition information to prune data files.

Each Iceberg write produces a new snapshot, and commit atomicity is achieved by renaming metadata files or through conditional updates to an external catalog, simplifying concurrency control from page-level locks to metadata-level optimistic concurrency.

Object storage such as S3 only guarantees the atomicity of a single-object PUT, with no cross-object rename or transactions, so Iceberg's atomic commits on object storage rely on mechanisms such as rename, Hive Metastore database transactions, or the CAS of a REST catalog.

Iceberg strips partition information out of the physical directory, with partitions existing as transformation expressions in metadata, so schema evolution does not require rewriting historical data.

InnoDB excels at high-frequency small transactions and low-latency point lookups, while Iceberg excels at massive batch writes, analytical scans and schema evolution.

InnoDB's complexity comes from the premise of 'in-place updates', while Iceberg's simplicity comes from the premise of 'immutable files'; when choosing a storage solution, one should first determine the write pattern and storage medium before deciding the metadata form.

Iceberg was initiated by Netflix and contributed to the Apache Foundation, with an emphasis on multi-engine compatibility (accessible by Spark, Trino, Flink, and Hive).

All three—Delta Lake, Iceberg, and Hudi—support ACID transactions, time travel (historical version queries), schema evolution, and partition pruning.

The lakehouse architecture is based on open storage formats (such as Parquet) and open table formats (such as Delta Lake, Iceberg, and Hudi), providing capabilities such as transactions, version management, and schema evolution on top of low-cost object storage.

Iceberg and Delta Lake use optimistic concurrency control mechanisms, where write operations commit transactions by atomically updating the pointer to the current metadata file.

Open table formats (such as Apache Iceberg and Delta Lake) persist the complete state of a table as metadata files in object storage, rather than relying on a centralized service for maintenance.

Apache Iceberg's table metadata layer is separated from data files, allowing catalogs to attach governance attributes without modifying the underlying data.

Iceberg's snapshot isolation mechanism provides historical version support for auditing.