Synthetic Data

Synthetic data refers to data artificially generated through algorithms rather than collected directly from the real world; it simulates the statistical characteristics of real data distributions but does not correspond to real individuals or events.

When language models are repeatedly trained on data generated by themselves or similar models, systematic degradation occurs, with the diversity of model outputs decreasing with each generation and the tail information of the distribution gradually being lost.

If the synthetic data generation process omits correlations between key variables, models trained on such data often perform poorly in real-world deployment environments.

Gartner predicted that by 2024, 60% of data used for AI development would be synthetic data.

MarketsandMarkets estimates that the synthetic data market will grow from approximately $300 million in 2023 to nearly $2 billion in 2028.

合成数据在长尾场景补充和隐私敏感场景替代两个维度上价值最高，而非作为主体数据源

A common approach for generating synthetic data is to have a model rewrite, expand, or role-shift based on real user conversations, producing training samples that are fictitious in form but retain the semantic information of the original.

Research published in 2023, such as "The Curse of Recursion," shows that when models are repeatedly trained on their own generated data, the data distribution gradually degrades, diversity continuously decreases, and model capabilities systematically decline (model collapse).

Procedural synthetic data can generate on demand edge-case scenarios that are difficult to collect in reality—such as arbitrary-intensity glare, low illumination, and heavy occlusion—and provides precise annotations.

Procedural synthetic data provides robotics and autonomous driving developers with a low-cost, high-precision, and infinitely scalable data acquisition path

Anomaly detection methods are friendlier to unknown defect types but have a relatively high false positive rate; synthetic data can expand sample size but has a domain gap from the real defect distribution.

The data scarcity trend is forcing AI teams to turn to three types of alternative data sources: user behavior logs and interaction records inside enterprises, proprietary data in vertical industry domains, and manually annotated or synthetically generated data.

Synthetic data struggles to reproduce the unpredictability, emotional nuances, and strategic decision-making in real human behavior.

A common reason for synthetic data failure is that the model learns synthesis-specific artifacts rather than real tampering features, leading to overfitting on the synthetic set and near-zero performance on the real test set.

An inherent limitation of the synthetic data framework is the gap between artificially constructed data and real-world complexity.

Laboratories filter and up-weight high-complexity use cases because they are most valuable for improving the model's reasoning and professional capabilities

Synthetic data does not directly use raw user input content, but rather new data derived, rewritten, or reconstructed from that data

According to the leak, the probability that data from high-value use cases—such as users working on complex mathematics, business analysis, software engineering, or biology problems—is incorporated into training is significantly increased

Synthetic data can fill gaps in real data regarding cost, privacy, and coverage in a controllable manner.

Companies such as NVIDIA have in recent years heavily invested in simulation training and synthetic data, having robots practice millions of times in virtual environments to compensate for the lack of real-world data.