The Small-File Explosion: Why Streaming Writes Bring Down Iceberg Lakehouses

Real-time lakehouses have been one of the hottest topics in data engineering over the past two years. Business teams demand that operational dashboards and smart analytics surface the latest data within seconds, so teams wire Flink or Kafka directly into cloud object storage and commit every 30 seconds. In test environments, this setup looks bulletproof—push one event in, and a few seconds later it shows up in a SQL query.

Production is a different story. Continuous commits manufacture a flood of roughly 120 KB Parquet fragment files scattered across dozens of partitions; within two months, a table that started out clean can accumulate more than 4 million individual objects. The Logic42 engineering team observed on customer sites that after a financial customer piped Flink continuously into an Iceberg lakehouse, the query engine simply gave up responding. The problem isn't the storage engine itself—it's the write pattern.

How small files destroy lakehouse query performance

When a distributed query engine executes a query, most of its time goes into reading object storage metadata, not scanning column values. When Trino or ClickHouse issues a query, it must first parse thousands of Iceberg manifest files, resolve object URLs, and then fire off a separate HTTP GET request for every small file.

Logic42 shared a set of performance figures that track file fragmentation, clearly showing the full journey from healthy to broken:

File size	Files per partition	Metadata snapshot size	S3 LIST/GET requests	Trino planning latency	End-to-end query time
512 MB (target)	4	140 KB	28	45 ms	480 ms
128 MB (acceptable)	16	520 KB	112	120 ms	1,250 ms
10 MB (degraded)	200	6.8 MB	1,400	850 ms	4,600 ms
1 MB (severe fragmentation)	2,000	64 MB	14,000	4,200 ms	18,900 ms
120 KB (streaming crisis)	18,500	4.8 GB	140,000	38,000 ms	Timeout (OOM)

Once you hit the streaming-crisis tier, a single dashboard refresh triggers 140,000 S3 GET calls, AWS throttling immediately responds with HTTP 503 SlowDown errors, redundant API costs alone run to $4,200 per month, and executive dashboards crash on out-of-memory errors.

The structure of Iceberg manifests and the metadata tree

To understand why small files bring down queries, you first need to understand Iceberg's layered table metadata structure. An Iceberg table doesn't point directly at data files; it maintains a multi-level metadata tree: at the top is the table snapshot, below that is the manifest list, below that are manifest files, and only at the bottom are the actual data files. Each manifest file records the paths of a batch of data files, their partition values, and per-column min/max statistics. During the planning phase, the query engine must read this entire tree to determine which data files are relevant to the WHERE clause. That means even if a query hits only a few rows, the engine still has to walk every manifest. When a table holds 4 million 120 KB files, the manifests themselves number in the thousands and total several GB—just loading that metadata tree into memory is enough to exhaust Trino's coordinator heap. Delete files and snapshot expiration markers also inflate this tree, so the cost of small files isn't just storage—it's a multiplier effect across the entire metadata system.

Three hidden costs of lakehouse fragmentation

Unmanaged streaming files don't just slow down ad-hoc queries—they erode the entire data platform at three levels.

1. Object storage metadata lock-in

S3 and Google Cloud Storage aren't hierarchical POSIX filesystems. When an engine lists 500,000 files in a single partition, it has to page through hundreds of metadata continuation tokens. The time spent reading manifest files is thirty times the time spent actually executing SQL filters.

2. Loss of columnar pruning and vectorization

Parquet's high performance comes from dictionary encoding and columnar compression. In a 512 MB file, column chunks compress efficiently, and min/max row-group statistics let readers skip entire blocks of data. But in a 120 KB file, the header and footer alone consume 40% of the bytes, and compression gains drop to nearly zero.

How Parquet row groups, dictionary encoding, and min/max pruning work

Parquet's efficient reads rely on three mechanisms working together. First are row groups, about 128 MB by default, each storing columns internally as independent column chunks. Second is dictionary encoding, where the engine builds a dictionary of repeated values in a column, replaces original strings with integer indices, and combines that with RLE compression—this can deliver 10x or greater compression ratios on low-cardinality columns. Third, each column chunk header records min/max statistics, which readers use to skip entire row groups that don't match the predicate, a process called predicate pushdown or data skipping. The larger the file, the wider the value range within a row group, the richer the dictionary, and the more discriminative the statistics. A 120 KB small file, by contrast, often holds only a few hundred rows, so the row-group count approaches 1, the dictionary barely gets built before it has to close out, and the footer's schema, statistics, and column-offset tables still consume about 40% of the bytes—compression gains hit zero. At that point the file degrades into something close to row-oriented, coarse-grained storage, and columnar pruning and vectorized scans are both lost.

3. Manifest tree churn and commit conflicts

Every streaming micro-batch generates a new table snapshot. When multiple Flink workers commit concurrently while unoptimized compaction jobs are also running, Iceberg's optimistic concurrency control triggers commit collisions. Batches fail, which in turn backpressures the Kafka brokers.

Iceberg optimistic concurrency control and snapshot isolation

Iceberg uses optimistic concurrency control to coordinate multiple writers. Each commit generates a new snapshot based on the current one, landed by atomically swapping the metadata pointer in the catalog; if the pointer is detected to have changed by another writer, the commit fails and triggers a retry. In streaming scenarios, dozens of parallel Flink subtasks each commit every 30 seconds, and background compaction jobs are also constantly rewriting files and committing snapshots, so the collision probability spikes. Failed retries re-read the latest snapshot and re-plan the write, which lengthens Kafka consumer lag, triggers Flink backpressure, and ultimately propagates the pressure back to the Kafka brokers. Snapshot isolation was designed as a safety mechanism to decouple reads from writes, but under high-frequency, small-batch commits it becomes a conflict amplifier. Reducing commit frequency, widening single-batch write granularity, and scheduling compaction in low-conflict windows are the direct ways to mitigate this.

Architecture comparison: direct streaming writes vs. bin-packing compaction

To preserve both sub-second query performance and real-time visibility, you have to decouple the streaming write buffer from the queryable analytical partitions. In their engineering practice, Logic42 deployed an asynchronous compaction pipeline that transparently merges streaming fragments without locking active readers.

The core approach is to run a standalone Spark or PyIceberg worker that continuously merges files smaller than 64 MB into optimal 512 MB columnar blocks, keeping Trino's file planning under 50 ms. Here's a snippet of compaction job code from their Sovereign Lakehouse:

# Logic42 Sovereign Lakehouse: Scheduled Iceberg Compaction Runner
from pyiceberg.catalog import load_catalog
from pyiceberg.table import Table

class SovereignIcebergOptimizer:
    def __init__(self, catalog_name: str, warehouse_uri: str):
        self.catalog = load_catalog(catalog_name, **{"warehouse": warehouse_uri})

    def execute_bin_packing(self, table_identifier: str, target_file_size_mb: int = 512):
        table: Table = self.catalog.load_table(table_identifier)
        target_bytes = target_file_size_mb * 1024 * 1024

        print(f"[OPTIMIZER] Analyzing snapshot lineage for {table_identifier}...")

        # Rewrite data files using bin-pack strategy with target byte boundaries
        rewrite_result = table.rewrite_data_files(
            strategy="binpack",
            options={
                "target-file-size-bytes": str(target_bytes),
                "min-file-size-bytes": str(int(target_bytes * 0.75)),
                "max-file-size-bytes": str(int(target_bytes * 1.25)),
                "max-concurrent-file-group-rewrites": "8"
            }
        )

        # Expire obsolete snapshots older than 3 days to reclaim object storage
        table.expire_snapshots(older_than_ms=3 * 86400 * 1000)

        print(f"[OPTIMIZER] Compaction completed successfully. Consolidated {rewrite_result.rewritten_data_files_count} files into {rewrite_result.added_data_files_count} clean blocks.")

Three ironclad rules for sovereign lakehouse writes

Logic42 distilled their experience into three directly actionable rules.

Decouple the write buffer from the analytical gold tables. Route high-speed streaming events into an append-only raw buffer or temporary staging partitions—don't write 30-second commits straight into core production datasets.

Automate continuous bin-packing compaction. Use a standalone worker to continuously merge files smaller than 64 MB into 512 MB columnar blocks, keeping Trino file planning under 50 ms.

Aggressively prune snapshot lineage. Retain only the table snapshots required by your rollback SLA. Cleaning up orphan manifests older than 72 hours can reclaim terabytes of ghost storage on private MinIO or AWS S3 buckets.

The small-file explosion is fundamentally an operational debt: the convenience you buy up front with real-time writes ultimately gets repaid with interest in the form of metadata bloat, cloud costs, and query timeouts. A real-time lakehouse that actually scales was never built by wiring Flink straight into object storage—it's built on a carefully designed compaction buffer layer between writes and queries.

One-line Summary

Streaming directly into object storage creates massive small files in Iceberg tables that cripple metadata and query performance.

Paragraph Summary

The article dissects a classic trap in real-time lakehouses: committing Flink or Kafka directly to Iceberg every 30 seconds produces a flood of roughly 120 KB Parquet fragment files across dozens of partitions, accumulating over 4 million objects within two months. Logic42's measured data shows that as files degrade from 512 MB to 120 KB, files per partition rise from 4 to 18,500, S3 LIST/GET requests climb from 28 to 140,000, Trino planning latency surges from 45 ms to 38 seconds, and end-to-end query time goes from 480 ms to timeout OOM. The costs show up in three places: hierarchical object storage metadata locks up, Parquet's columnar pruning and dictionary compression gains are lost, and Iceberg manifest tree churn triggers commit conflicts and Kafka backpressure. The solution the article offers is to decouple the streaming write buffer from queryable analytical partitions and use a standalone worker to continuously bin-pack files under 64 MB into 512 MB blocks, while aggressively pruning snapshots and orphan manifests—framing the whole thing as operational debt that must be repaid up front.