FineWeb-Edu
A high-quality web text dataset released by Hugging Face in 2024. It is filtered from CommonCrawl for educational content, retaining high-quality samples via classifier scoring, and is suitable for training small language models.
Timeline (last 90 days)
The model is trained in two stages: first masked language model pre-training on about 20 million sentences from FineWeb-Edu, then correction fine-tuning on 12.02 million (noisy, clean) pairs, of which 30% are identity pairs.
The model trained using Edu-QuRating-filtered mixed sets achieved a higher overall observed accuracy than the FineWeb-Edu baseline model across nine benchmark tests, with performance gains concentrated on specific tasks.
HuggingFace released the FineWeb-Edu dataset in 2024, scoring the educational value of web page content on a scale of 0–5 by training a classifier.
In controlled single-run pre-training comparison experiments, the model trained on the Edu-QuRating-curated mixture achieved higher overall observed accuracy across nine benchmarks than the FineWeb-Edu baseline model, with performance gains concentrated on specific tasks.
All Facts (4)
The model is trained in two stages: first masked language model pre-training on about 20 million sentences from FineWeb-Edu, then correction fine-tuning on 12.02 million (noisy, clean) pairs, of which 30% are identity pairs.
50%UnverifiedThe model trained using Edu-QuRating-filtered mixed sets achieved a higher overall observed accuracy than the FineWeb-Edu baseline model across nine benchmark tests, with performance gains concentrated on specific tasks.
50%UnverifiedHuggingFace released the FineWeb-Edu dataset in 2024, scoring the educational value of web page content on a scale of 0–5 by training a classifier.
50%UnverifiedIn controlled single-run pre-training comparison experiments, the model trained on the Edu-QuRating-curated mixture achieved higher overall observed accuracy across nine benchmarks than the FineWeb-Edu baseline model, with performance gains concentrated on specific tasks.
50%