OpenWebText
A commonly used text dataset for language model training and evaluation.
Timeline (last 90 days)
On the OpenWebText dataset, recurrent UDLM reaches the same validation NLL as non-recurrent UDLM using only about 1/2.8 of the FLOPs corresponding to the non-recurrent model's 0.5 trillion-token training budget.
On language modeling tasks (Text8, TinyStories, OpenWebText), binary diffusion remains competitive but does not significantly surpass the baselines.
GPT-2's validation loss on the OpenWebText dataset is approximately 3.28, corresponding to a perplexity of approximately 26.6.
All Facts (3)
On the OpenWebText dataset, recurrent UDLM reaches the same validation NLL as non-recurrent UDLM using only about 1/2.8 of the FLOPs corresponding to the non-recurrent model's 0.5 trillion-token training budget.
50%UnverifiedOn language modeling tasks (Text8, TinyStories, OpenWebText), binary diffusion remains competitive but does not significantly surpass the baselines.
50%UnverifiedGPT-2's validation loss on the OpenWebText dataset is approximately 3.28, corresponding to a perplexity of approximately 26.6.
50%