Iceberg
An open-source high-performance table format from Apache that provides data lakes with ACID transactions, schema evolution, and partition management. It is an open format standard in data lakehouse architecture, alongside Delta Lake.
Timeline (last 90 days)
The direct means of mitigating Iceberg streaming write commit conflicts include: reducing commit frequency, increasing the granularity of each batch write, and scheduling compaction in low-conflict windows.
Continuously streaming writes into an Iceberg lakehouse (e.g., committing once every 30 seconds) creates large numbers of roughly 120 KB Parquet fragment files across dozens of partitions, and within two months a single table can accumulate more than 4 million independent objects.
The Logic42 engineering team observed a financial customer whose query engine simply gave up responding because Flink was continuously writing into an Iceberg lakehouse.
Iceberg table metadata uses a multi-level tree structure: table snapshot -> manifest list -> manifest file -> data file, and the query engine must fully read the entire metadata tree during the planning phase.
When a table contains 4 million 120 KB small files, the manifest files themselves can number in the thousands and total several GB, and merely reading this metadata tree can exhaust the Trino coordinator's heap space.
Iceberg uses optimistic concurrency control to coordinate multiple writers: at commit time it atomically replaces the metadata pointer in the catalog, and if the pointer has already been changed by another writer the commit fails and is retried.
In streaming scenarios, dozens of Flink parallel subtasks committing a snapshot every 30 seconds while background compaction tasks are layered on top cause the conflict probability of Iceberg's optimistic concurrency control to rise sharply, which in turn triggers backpressure that propagates to the Kafka broker.
The lakehouse architecture combines a data lake in open formats (such as Parquet and Iceberg) with the transactional and query capabilities of a database, allowing the same data to be accessed by both the Postgres engine and analytical engines such as Spark and Trino.
All Facts (8)
The direct means of mitigating Iceberg streaming write commit conflicts include: reducing commit frequency, increasing the granularity of each batch write, and scheduling compaction in low-conflict windows.
50%UnverifiedContinuously streaming writes into an Iceberg lakehouse (e.g., committing once every 30 seconds) creates large numbers of roughly 120 KB Parquet fragment files across dozens of partitions, and within two months a single table can accumulate more than 4 million independent objects.
50%UnverifiedIceberg uses optimistic concurrency control to coordinate multiple writers: at commit time it atomically replaces the metadata pointer in the catalog, and if the pointer has already been changed by another writer the commit fails and is retried.
50%UnverifiedIn streaming scenarios, dozens of Flink parallel subtasks committing a snapshot every 30 seconds while background compaction tasks are layered on top cause the conflict probability of Iceberg's optimistic concurrency control to rise sharply, which in turn triggers backpressure that propagates to the Kafka broker.
50%UnverifiedIceberg table metadata uses a multi-level tree structure: table snapshot -> manifest list -> manifest file -> data file, and the query engine must fully read the entire metadata tree during the planning phase.
50%UnverifiedWhen a table contains 4 million 120 KB small files, the manifest files themselves can number in the thousands and total several GB, and merely reading this metadata tree can exhaust the Trino coordinator's heap space.
50%UnverifiedThe lakehouse architecture combines a data lake in open formats (such as Parquet and Iceberg) with the transactional and query capabilities of a database, allowing the same data to be accessed by both the Postgres engine and analytical engines such as Spark and Trino.
50%UnverifiedThe Logic42 engineering team observed a financial customer whose query engine simply gave up responding because Flink was continuously writing into an Iceberg lakehouse.
50%