Data Wall
A concept in the pretraining domain referring to the approaching exhaustion of high-quality human text available for training large models, causing the marginal returns of the traditional approach of scaling up crawling to decline rapidly and prompting frontier laboratories to turn to data curation and synthetic data.
Timeline (last 90 days)
In its pretraining report, Microsoft emphasized the role of data curation and synthetic data in breaking through the data wall.
Research institutions such as Epoch AI estimate that large models may face a "data wall" problem within the next few years, meaning high-quality data will be fully utilized.
The AI research community refers to the dilemma of limited high-quality publicly available text data on the internet as model scale increases as the "Data Wall".
All Facts (3)
In its pretraining report, Microsoft emphasized the role of data curation and synthetic data in breaking through the data wall.
50%UnverifiedThe AI research community refers to the dilemma of limited high-quality publicly available text data on the internet as model scale increases as the "Data Wall".
50%UnverifiedResearch institutions such as Epoch AI estimate that large models may face a "data wall" problem within the next few years, meaning high-quality data will be fully utilized.
50%