Codex was fine-tuned on top of GPT-3 using approximately 154 billion tokens of publicly available code repository data covering 54 programming languages, achieving a pass@1 accuracy of 28.8% on the HumanEval benchmark, compared to 0% for GPT-3 at the same time.