5.6 Billion TikTok Video Metadata Records Open-Sourced on Hugging Face

A developer recently shared a remarkably large open-source dataset on Reddit: 5.6 billion TikTok video metadata records, now available on Hugging Face. The dataset spans from 2014 all the way to October 2026, covering nearly the entire lifecycle of the TikTok platform. For anyone studying social media behavior, content virality, or recommendation algorithms, this is an unusually large and comprehensive real-world sample.

Dataset Scale and Structure

By any measure in the public social data space, this dataset is exceptional. It consists of three core tables, each corresponding to a different entity within the TikTok ecosystem:

• Creators table: ~4.5 billion rows, containing account-level information for a massive number of platform creators.
• Videos table: ~5.6 billion rows, the centerpiece of the dataset, holding video-level metadata.
• Sounds table: ~633 million rows, covering the background music and audio clips that are central to TikTok's identity.

This three-table structure cleanly reflects the three pillars of TikTok's content ecosystem: people (creators), content (videos), and the medium that drives virality (sound). A huge part of TikTok's viral mechanics relies on the same audio clip being reused across thousands or millions of videos, so having a dedicated Sounds table with over 600 million rows offers unique value for analyzing content propagation chains.

It's worth emphasizing that this is metadata, not the video files themselves. The dataset contains descriptive, structured fields about videos — such as publish time, associated creator, and audio used — rather than playable media. This significantly reduces storage requirements and also sidesteps the risks of directly distributing copyrighted content.

Two Access Methods: Download and Online Query

Given the sheer scale of 5.6 billion rows, downloading everything isn't practical for everyone. The author offers two paths.

The first is the full dataset available through Hugging Face (datasocial/tiktok-5.6B-videos), suitable for teams with sufficient storage and compute who need offline batch processing.

The second is a ClickHouse database the author set up themselves, allowing interested users to run queries directly online without pulling billions of rows locally. ClickHouse is a columnar database designed for analytical workloads — it excels at aggregation queries over massive datasets, making it a sensible choice as a query interface. The author mentioned in the post that database credentials can be shared via direct message with interested users.

That said, the author made a clear request: since the database is self-hosted, users should avoid running heavy queries that could overwhelm the server. This seemingly casual note actually highlights a real challenge with personal open-source projects like this — the data is compelling, but the maintenance costs and reliability all fall on one person.

---

ClickHouse was open-sourced by the Russian company Yandex in 2016 and is designed specifically for OLAP (Online Analytical Processing) workloads. Unlike traditional row-oriented databases, columnar storage means queries only read the columns they need rather than entire rows — this can deliver orders-of-magnitude performance gains for aggregation operations like "sum total likes for a category of videos." ClickHouse also supports vectorized execution and data compression, enabling second-level response times even when processing hundreds of billions of rows on a single machine. This is why companies like ByteDance and Cloudflare use it for internal log analysis and user behavior analytics. For this TikTok dataset specifically, ClickHouse's compression capabilities also help store billions of records within a reasonable footprint, without requiring storage equivalent to the raw uncompressed data.

---

A Resource That Comes With Real Concerns

From a research standpoint, the appeal of this kind of dataset is obvious. Complete social platform metadata is invaluable for studying content virality, creator growth trajectories, and audio-driven trend cycles — data that platforms typically keep tightly locked away, making large-scale external research access extremely difficult.

But the concerns surrounding this dataset are hard to ignore. The first is compliance around data sourcing. TikTok does not offer any public API for bulk data access at this scale. 5.6 billion video metadata records and 4.5 billion creator records almost certainly originated from large-scale scraping, which places this squarely in the gray zone of platform terms of service and potentially applicable law.

The second concern is privacy. Even as metadata, 4.5 billion creator records may contain substantial amounts of user-linked information. Publicly distributing this data carries real risks of impacting individual privacy. Researchers should carefully assess the ethical and legal boundaries before using it.

Notes for Developers and Researchers

If you're planning to use this dataset, a few things are worth keeping in mind: verify what the specific fields mean and how complete the data is; assess whether your intended use case is consistent with the source platform's terms of service; and follow privacy protection principles in any analysis involving personal data. For those who just want to quickly validate an idea, the author's online query interface is a much lower-cost starting point — though be mindful of the limits of a personally hosted server.

The existence of this dataset once again reflects the open-source community's appetite for large-scale real-world data, while also highlighting the persistent tension around how such data is obtained and whether it's compliant. It's both a resource worth paying attention to and one that deserves a cautious approach.

---

The legal boundaries around large-scale web scraping vary significantly across jurisdictions. In the US, the Ninth Circuit Court of Appeals ruled in the 2022 hiQ Labs v. LinkedIn case that scraping publicly accessible data does not violate the Computer Fraud and Abuse Act (CFAA) — but this does not mean scraping is entirely legal. A platform's Terms of Service can still create contractual liability, and the EU's General Data Protection Regulation (GDPR) imposes much stricter requirements on the handling of data involving European users. If this dataset's 5.6 billion creator and video records contain personally identifiable fields, distributing and using it within the EU would face substantial compliance challenges. This explains why datasets of this kind are often released under the banner of "research use" — but the responsibility for legal risk assessment still falls squarely on individual users.