Data Wall
A concept in the pretraining domain referring to the approaching exhaustion of high-quality human text available for training large models, causing the marginal returns of the traditional approach of scaling up crawling to decline rapidly and prompting frontier laboratories to turn to data curation and synthetic data.
Core Facts
Timeline (last 90 days)
The data wall refers to the stock of high-quality public text approaching exhaustion, as the training of existing mainstream large models has consumed most of the high-quality corpus available on the internet.
In its pretraining report, Microsoft emphasized the role of data curation and synthetic data in breaking through the data wall.
Research institutions such as Epoch AI estimate that large models may face a "data wall" problem within the next few years, meaning high-quality data will be fully utilized.
According to estimates by Epoch AI research, high-quality text data available on the internet for training will be largely exhausted by around 2026.
The AI research community refers to the dilemma of limited high-quality publicly available text data on the internet as model scale increases as the "Data Wall".
All Facts (5)
According to estimates by Epoch AI research, high-quality text data available on the internet for training will be largely exhausted by around 2026.
65%UnverifiedThe data wall refers to the stock of high-quality public text approaching exhaustion, as the training of existing mainstream large models has consumed most of the high-quality corpus available on the internet.
50%UnverifiedIn its pretraining report, Microsoft emphasized the role of data curation and synthetic data in breaking through the data wall.
50%UnverifiedThe AI research community refers to the dilemma of limited high-quality publicly available text data on the internet as model scale increases as the "Data Wall".
50%UnverifiedResearch institutions such as Epoch AI estimate that large models may face a "data wall" problem within the next few years, meaning high-quality data will be fully utilized.
50%