WikiText-103
A large-scale language modeling dataset extracted from Wikipedia, containing over 100 million tokens. It is widely used for evaluating language model perplexity and for pretraining research, serving as a standard benchmark across a broad range of natural language processing studies.
Timeline (last 90 days)
After replacing the intermediate attention modules with OAL and jointly fine-tuning via LoRA, the WikiText-103 perplexity dropped from 31.71 to 27.50 compared to the global quadratic kernel.
The study validates on three datasets—TinyStories, WikiText-103, and CNN/DailyMail—scaling each corpus from 512 chunks to 2,048 chunks, with each chunk consisting of 64 tokens.
WikiText-103由Merity等人于2016年从英文维基百科提取构建，包含约1.03亿token（约28,000篇完整文章）
在WikiText-103数据集上，约10M参数下CWAA V6验证困惑度为139.09，标准Transformer为149.19，CWAA约有7%相对改善
WikiText-103是包含约1亿token的英文维基百科语料，是语言模型基准测试的标准数据集之一
All Facts (5)
After replacing the intermediate attention modules with OAL and jointly fine-tuning via LoRA, the WikiText-103 perplexity dropped from 31.71 to 27.50 compared to the global quadratic kernel.
50%UnverifiedThe study validates on three datasets—TinyStories, WikiText-103, and CNN/DailyMail—scaling each corpus from 512 chunks to 2,048 chunks, with each chunk consisting of 64 tokens.
50%UnverifiedWikiText-103由Merity等人于2016年从英文维基百科提取构建，包含约1.03亿token（约28,000篇完整文章）
50%Unverified在WikiText-103数据集上，约10M参数下CWAA V6验证困惑度为139.09，标准Transformer为149.19，CWAA约有7%相对改善
50%UnverifiedWikiText-103是包含约1亿token的英文维基百科语料，是语言模型基准测试的标准数据集之一
50%