Scraping 5.9 Billion TikTok Videos in 3 Weeks: A Deep Dive into Mobile Reverse Engineering and Large-Scale Data Collection

A developer recently posted on Reddit claiming to have scraped 5.94 billion TikTok videos and 3.23 billion user profiles in just three weeks, uploading the complete dataset for free to the open-source platform Hugging Face. The sheer scale and speed of this data collection effort has sparked widespread discussion in the tech community around large-scale web scraping, mobile API reverse engineering, and data compliance.

Reddit post: 5.9 billion TikTok videos and 3.23 billion user profiles scraped in 3 weeks

5.9 Billion Records: What's Actually in the Dataset

According to the poster, the dataset goes far beyond just videos — it encompasses nearly every type of publicly accessible content on the TikTok platform:

5.94 billion video metadata entries
3.23 billion user profiles
Comments and replies
Hashtags
Audio/sounds
And many more structured fields

The complete dataset has been published on Hugging Face (kuben-developer/tiktok-videos-4b) and is described as fully open-source and freely accessible. For researchers, a social media dataset at this scale is exceptionally rare — it could be used for content trend analysis, recommendation system research, social network graph construction, and even large-scale natural language and multimodal model training.

What's particularly remarkable is that collecting nearly 10 billion records in three weeks almost certainly required a highly parallelized distributed crawling architecture with efficient request scheduling. The speed alone is a significant engineering achievement.

The Core Approach: Reverse Engineering TikTok's Mobile App

The poster revealed that the foundation of this data collection capability is a TikTok mobile app reverse engineering method he developed several years ago.

Why Target the Mobile App Instead of the Web?

Compared to traditional web scraping, reverse-engineering a mobile app's API typically offers several distinct advantages:

Richer data: Mobile APIs tend to return more complete, structured fields than their web counterparts.
Weaker anti-scraping measures: Unlike the web, which involves complex front-end rendering and risk controls, mobile API responses are clean JSON.
Bypasses login requirements: Crucially — according to the developer, TikTok's app exposes 24 endpoints that are accessible without a logged-in account.

In other words, this data is technically "publicly accessible." The developer simply analyzed the communication protocol between the app and its servers, identified these unauthenticated endpoints, and automated their calls at massive scale.

The Typical Reverse Engineering Workflow

While the poster hasn't publicly released all technical details (the full code requires payment), based on his description, the general process for this type of mobile reverse engineering typically includes:

Traffic interception: Capturing the app's network requests using a man-in-the-middle proxy (e.g., mitmproxy, Charles).
Signature cracking: Apps like TikTok typically sign request parameters (e.g., X-Gorgon, X-Khronos). Reconstructing the signing algorithm is the core technical challenge.
Endpoint replication: Simulating legitimate requests outside the app environment to reliably call those 24 public endpoints.
Scaled orchestration: Using proxy pools, rate limiting, and distributed deployment to achieve billion-record collection throughput.

Cracking the signature mechanism is usually the most time-consuming and technically demanding part of this work — which explains why the author chose to charge for the code.

A Note on TikTok's Signing Mechanism

TikTok's X-Gorgon and X-Khronos headers form a proprietary request integrity verification system. X-Khronos is essentially the current Unix timestamp, while X-Gorgon is a signature string generated from a combination of request parameters, the timestamp, device fingerprints, and other fields via a private hashing algorithm. The server validates this signature on receipt to distinguish legitimate app traffic from automated scripts. Reversing this mechanism typically requires static disassembly or dynamic instrumentation of the app's native binaries (.so shared libraries on Android) to extract the signing function's implementation logic. This is why packet capture alone isn't sufficient for replication — captures only reveal the request output, while the signature generation logic lives inside the client-side code.

The Business Logic: Free Dataset, Paid Code

This project employs an interesting tiered strategy:

The dataset is completely free and open-source — uploaded to Hugging Face for anyone to download and use.
The collection code requires payment — the author offers the complete code and documentation on his website (tiktok-api.seeksocial.io) for a fee.

This "free output, paid tooling" model is quite clever. For the vast majority of researchers and data analysts, what they actually need is the data itself, not to reinvent the wheel. Publicly releasing the dataset for free builds technical credibility and expands reach, while also funneling traffic toward the paid code service — which targets commercial users who want to run their own continuous, real-time data collection.

The Unavoidable Data Compliance and Privacy Controversy

As impressive as the technical capability is, this project sits squarely in a legal and ethical gray zone. The poster himself acknowledged this with an honest disclaimer:

"TikTok's app exposes 24 endpoints accessible without an account, so the data is technically publicly accessible. However, accessing it this way likely still violates TikTok's Terms of Service (ToS)."

Several issues here deserve serious attention:

"Publicly Accessible" Is Not the Same as "Freely Scrapable"

Data being technically accessible is a different matter from it being legally permissible to collect at scale. Recent case law (such as hiQ v. LinkedIn) shows that the legality of scraping public data is highly dependent on the specific jurisdiction, the intended use, and whether any technical protection measures were circumvented.

On hiQ v. LinkedIn

hiQ Labs, Inc. v. LinkedIn Corp. is currently the most influential U.S. case on the legality of scraping publicly accessible data. After multiple rounds of appeals, the core dispute centered on whether automated scraping of public web data constitutes "unauthorized access" under the Computer Fraud and Abuse Act (CFAA). The Ninth Circuit's 2022 ruling leaned toward finding that scraping publicly available data (requiring no login) does not constitute unauthorized access under the CFAA — but this conclusion applies only within specific jurisdictions, does not exempt parties from civil liability for violating platform Terms of Service, and does not address independent compliance obligations under the EU's GDPR framework based on the principle of data minimization.

The Privacy Risks of Aggregating Large-Scale User Data

The open release of 3.23 billion user profiles means vast amounts of personal information have been aggregated. Even if each individual record is publicly available on its own, large-scale aggregation can pose a substantial threat to user privacy and may implicate data protection regulations such as GDPR and CCPA.

On Re-identification Risk

Re-identification is one of the most critical privacy risks associated with large-scale aggregation of public data. Even if a dataset contains no direct identifiers like names or ID numbers, cross-referencing fields such as usernames, profile pictures, geolocation tags, and posting time patterns can often restore anonymous records to real individuals with high confidence. GDPR Article 4 defines personal data as any information relating to an "identifiable natural person" — not just direct identifiers. As a result, even if the individual fields in 3.23 billion user profiles appear innocuous, the aggregated collection as a whole is very likely to constitute a body of personal data requiring legal protection. Both the publisher and users of such a dataset may bear corresponding data controller responsibilities.

Secondary Liability for Dataset Users

Researchers who download and use this dataset may also face compliance risks. Before applying it to any formal research or commercial use, careful evaluation of the data's legal provenance is strongly advised.

Conclusion: The Boundary Between Technical Capability and Responsibility

This project is an impressive feat of engineering — it demonstrates that through mobile reverse engineering, an individual developer can build a billion-record data collection capability in a remarkably short time. It represents a tempting resource for the research community, and once again highlights the ongoing challenges large platforms face in data protection.

But with greater technical capability comes the need for clearer boundaries of responsibility. How to strike a balance between the open-source ethos, the value of data research, user privacy, and platform terms of service is a question every data practitioner needs to think carefully about. For datasets like this one, our recommendation is simple: appreciate the engineering — but approach its use with caution.