Parquet

Delta Lake is a storage layer format open-sourced by Databricks, built on top of Parquet files, and provides features such as ACID transactions, version rollback, and Schema evolution.

A table format maintains a metadata layer on top of immutable open files such as Parquet and ORC (recording which files make up the table, the schema history, and the snapshot of each commit), enabling atomic commits, schema evolution, and time travel without modifying the underlying files.

The Lakehouse concept is to store data in open file formats (such as Parquet, ORC) while layering on data warehouse capabilities such as transactions, schema evolution, and time travel, avoiding the data redundancy and synchronization latency caused by running a data lake and a data warehouse as two parallel systems.

In a 120 KB small Parquet file, the file header and footer (the footer's schema, statistics, and column offset table) account for about 40% of the total bytes, and the compression gains are almost entirely nullified.

Continuously streaming writes into an Iceberg lakehouse (e.g., committing once every 30 seconds) creates large numbers of roughly 120 KB Parquet fragment files across dozens of partitions, and within two months a single table can accumulate more than 4 million independent objects.

Parquet's efficient reads rely on the coordination of three layers of mechanisms: row groups (about 128 MB by default), dictionary encoding and RLE compression, and data skipping driven by column chunk min/max statistics.

The platform adopts a medallion lakehouse model, with Bronze, Silver, and Gold layers based on Parquet, PostgreSQL as the serving layer, and about 456 data contracts guarding the data.

A data lake typically refers to a system built on low-cost object storage (such as Amazon S3) at its foundation, storing data in columnar formats such as Parquet or ORC.

Parquet's default write method disrupts Xet's deduplication logic, because its columnar encoding, Row Group boundaries, and Footer are rewritten each time, causing the byte layout to change dramatically.

For a 100MB Parquet file with 3,000 rows added near the beginning, nearly the entire file must be re-uploaded with default settings, whereas enabling content-defined chunking requires uploading only approximately 6 chunks.

The lakehouse architecture is based on open storage formats (such as Parquet) and open table formats (such as Delta Lake, Iceberg, and Hudi), providing capabilities such as transactions, version management, and schema evolution on top of low-cost object storage.

Parquet, as the underlying columnar storage format, has become the universal physical storage choice for these table formats thanks to its high compression ratio and support for vectorized reads.

The dataset's structured data is provided in Parquet format, and videos use the webdataset format.

Parquet is an open-source column-oriented storage format maintained by the Apache Software Foundation; when reading specific fields, only the corresponding columns need to be scanned, reducing I/O overhead.

If an engine bypasses the governance layer's row filtering when reading Parquet files, permission leakage may occur.

The lakehouse architecture combines a data lake in open formats (such as Parquet and Iceberg) with the transactional and query capabilities of a database, allowing the same data to be accessed by both the Postgres engine and analytical engines such as Spark and Trino.

Delta tables are based on the Delta Lake format, which adds a transaction log on top of Parquet files to support ACID transactions, Time Travel, Schema Enforcement, and efficient insert/delete/update operations.