Same Job, Two Engines: Comparing InnoDB and Iceberg Internals

When the Same Job Is Handed to Two Engines

In the world of data, one class of problem keeps recurring: how do you efficiently store, index, and query massive amounts of structured data? MySQL's InnoDB and Apache Iceberg give completely different answers. The former is a row-based storage engine that has served online transactions for decades; the latter is an open table format born for the data lake. A discussion titled "Same Job, Two Engines: InnoDB and Iceberg Internals" from an RSS feed puts these two seemingly parallel systems on the same operating table for dissection.

They Are Really Solving the Same Problem

When people put InnoDB and Iceberg side by side, the first reaction for many is: "One is a database engine, the other is a table format—what is there to compare?" That division comes from their usage scenarios, not from the problems they solve. Peel away the layer of scenario, and their objectives overlap heavily:

• Provide a reliable data structure on top of unreliable storage;
• Use metadata to describe how data files are organized;
• Let query engines read only the portion of data they actually need;
• Perform concurrency control between writes and reads.

InnoDB accomplishes this with B+ tree pages, redo log, undo log, and MVCC; Iceberg accomplishes this with manifest files, manifest lists, snapshots, and metadata JSON. The task is the same, but the implementation paths are almost opposite at every turn.

InnoDB Internals: Built for Online Transactions

InnoDB's design philosophy can be summed up as "state resides in memory and on-disk pages, and changes rely on log-first writes."

Pages, B+ Trees, and Clustered Indexes

Data is organized into 16KB pages, and the table itself is a B+ tree ordered by primary key—that is, a clustered index. This means a primary key lookup is almost a single page localization, while a secondary index must go back to the table. Row data, indexes, and transaction information are all packed into the same page structure, making the read/write path very short.

Clustered index and secondary index table lookup are key to understanding InnoDB's read/write costs. A clustered index means the row data itself is stored in the leaf nodes of the B+ tree in primary key order, so a primary key point lookup only needs to navigate from the root page down to the leaf page to directly retrieve the entire row. A secondary index's leaf nodes store only the indexed column values and the primary key value. If the queried columns go beyond what the index covers, the obtained primary key must be used to look up the full row in the clustered index again—a process called "table lookup." The cost of table lookup lies in an extra B+ tree traversal and possibly random I/O. To reduce table lookups, InnoDB supports covering indexes: when all columns required by the query are contained in the secondary index, results can be returned directly without accessing the clustered index. This also explains why primary key length affects the size of all secondary indexes—each secondary index entry carries the primary key value.

Log-First Writes and Crash Recovery

Any modification is first written to the redo log, then dirty pages are flushed asynchronously. After a crash, recovery is done by replaying redo, and rollback and multi-version reads are achieved through undo. InnoDB clearly separates "data files" from "modification logs," trading space complexity for stable latency in online transactions.

MVCC and Concurrency

InnoDB's multiple versions are not stored in separate files; instead, rows are modified in place and old versions are tracked with undo chains. Read operations determine which version to see based on transaction visibility. This design makes single-row updates very fast, at the cost of coupling historical versions with current data, requiring a purge thread for cleanup.

The core goal of MVCC (multiversion concurrency control) is to let reads not block writes and writes not block reads, thereby maintaining high concurrency while ensuring isolation. In InnoDB, each row record hides a transaction ID and a rollback pointer; the rollback pointer links old versions in the undo log to form a version chain. When a read transaction accesses a row, it follows the version chain to find the version visible to the current transaction: Read Committed (RC) generates a new snapshot each time a statement executes, while Repeatable Read (RR) generates a snapshot at the start of the transaction. Visibility is determined by comparing the transaction ID with the Read View. Old versions do not disappear automatically; the purge thread asynchronously cleans them up after confirming that no earlier active transaction references them. This "in-place update + undo chain" approach keeps single-row write latency extremely low, but long transactions can hold back purge, causing undo space to bloat.

Iceberg Internals: Rebuilding Table Semantics for the Data Lake

Iceberg faces a completely different problem: data sits on S3 or HDFS, object storage is immutable, has no transactions, and has no indexes. Its solution is to extract all mutable meta-information and place it into layered metadata files.

Layered Metadata

The table metadata JSON points to a manifest list, the manifest list points to multiple manifest files, and each manifest file records the path, partition, column-level statistics, and file-level statistics of each data file. During query planning, metadata is read first, and partitions and statistics are used to prune files that do not need to be read.

Manifest files and column-level statistics are the cornerstone of Iceberg's query pruning. Each manifest file is an Avro file containing entries for several data files. Each entry includes the file path, its partition, record count, file size, and statistics such as the upper and lower bounds and null counts for each column. During query planning, Iceberg first reads the table metadata to locate the manifest list, then skips irrelevant manifests based on partition transform results and query predicates, and finally uses column-level min/max inside manifests to further eliminate data files. This "metadata first, data second" order means an analytical query may scan only a few percent of the total data volume. The accuracy of statistics directly determines pruning effectiveness, so Iceberg also provides index formats such as Puffin to supplement Bloom filters or finer-grained statistics.

Snapshots and Atomic Commits

Each write produces a new snapshot, while old snapshots preserve history. The atomicity of commits is achieved through "renaming metadata files" or conditional updates in an external catalog. This is equivalent to replacing InnoDB's page-level concurrency control with a single pointer switch.

Atomic commits on object storage are the core engineering challenge Iceberg faces. Object storage such as S3 guarantees atomicity only for single-object PUT, with no cross-object rename or transaction. Iceberg's solution is: each write first generates new manifests and a manifest list, then generates a new table metadata JSON, and finally switches the "current metadata pointer" in the catalog to the new file through a single atomic operation. There are three common implementations: Hive Metastore uses database transactions for conditional updates, REST catalog uses server-side CAS, and pure file system solutions rely on the atomicity of rename. Old snapshots are not deleted immediately, but retained in metadata for time travel queries until expire snapshots is explicitly executed. This "write new files, switch pointer" pattern simplifies concurrency control from page-level locks to metadata-level optimistic concurrency.

Hidden Partitioning and Schema Evolution

Iceberg strips partition information out of physical directories, so users no longer depend on Hive-style directory structures like dt=2024/region=cn. Partitions can exist as transform expressions in metadata, and schema evolution does not require rewriting historical data.

Comparison Table: Two Solutions to the Same Task

Dimension	InnoDB	Iceberg
Data organization	Row-based pages + B+ tree	Columnar files + metadata tree
Concurrency control	Row locks + undo + MVCC	Snapshot isolation + atomic commits
Change log	redo / undo	No logical log; relies on new snapshots
Read path optimization	Indexes + buffer pool	File-level predicate pushdown + statistics pruning
Historical versions	undo chain	Old snapshots + manifests
Storage assumption	Local or block storage	Object storage

This table also explains what each excels at: InnoDB is good at high-frequency small transactions and low-latency point lookups, while Iceberg is good at massive batch writes, analytical scans, and schema evolution.

Why Comparison Is More Valuable Than Understanding Each Alone

Looking at InnoDB alone, you might think "a table must have row locks and a redo log"; looking at Iceberg alone, you might think "a table is just a set of files plus metadata." Only by putting them together can you clearly see which constraints are essential and which are merely scenario choices.

InnoDB's complexity comes from the premise of "in-place updates"—as long as in-place modification is allowed, there must be CRUD logging and rollback mechanisms. Iceberg's simplicity comes from the premise of "immutable files"—as long as each write produces a batch of new files, commit degenerates into a pointer switch.

For engineers, this comparison offers two levels of insight. The first is at the design level: when choosing a storage solution, first determine the write pattern and storage medium, then decide the metadata form. The second is at the migration level: moving online business from InnoDB to Iceberg is not just changing a connection string, but redefining the trade-offs among consistency, latency, and cost.

Conclusion

For the same problem, InnoDB writes the standard answer for OLTP with pages, logs, and locks, while Iceberg writes the answer for the data lake with metadata, snapshots, and immutable files. Understanding their internals is not about deciding which is superior, but about knowing which constraint you are compromising with when facing a concrete task.

One-line Summary

InnoDB and Iceberg face the same storage and indexing problem but take opposite implementation paths due to different scenario constraints.

Paragraph Summary

The article dissects MySQL's InnoDB and the data lake table format Iceberg under the same framework, pointing out that although one is a database engine and the other is a table format, both solve the same problem: storage, indexing, metadata description, and concurrency control. InnoDB implements online transactions with 16KB pages, clustered B+ trees, redo/undo logs, and MVCC, with complexity stemming from the premise of "in-place updates"; Iceberg rebuilds table semantics on immutable object storage with table metadata, manifest lists, manifests, snapshots, and atomic commits, with simplicity stemming from the premise of "immutable files." Through a comparison table, the article shows their opposite choices in data organization, concurrency control, change logs, read path optimization, and historical versions, and emphasizes that the value of comparison lies in distinguishing essential constraints from scenario choices, thereby guiding storage selection and migration decisions.