Keera
A 97.5M-parameter language model for Python code with an architecture similar to GPT-2 small, trained from scratch by its developer, including a self-developed tokenizer and pretraining and fine-tuning pipelines.
Timeline (last 90 days)
A developer built a 97.5M-parameter Python code model called Keera from scratch, with a total training cost of $1.40.
Keera's architecture is based on GPT-2 small: 12 Transformer blocks, hidden dimension 768, and a total parameter count of 97.5M.
Keera uses a custom tokenizer with a vocabulary size of 16,384, trained on Python code, at an average of 3.9 characters/token.
Keera uses RoPE positional encoding, RMSNorm, weight-tied embeddings, and a 1,024-token context.
Keera was pretrained on the 2B-token codeparrot-clean dataset, following the Chinchilla rule of about 20 tokens/parameter.
Renting an RTX 5090 on Vast.ai ($0.584 per hour), using bf16 and torch.compile, Keera's pretraining throughput was about 234,000 tokens/second, the 2B-token training took 2 hours 23 minutes, and the validation loss was 1.501.
Keera was fine-tuned on the self-oss-instruct-sc2-exec-filter-50k dataset, which contains 50k Python instructions and answers, and all answer code was actually run and passed the tests.
On HumanEval, Keera's pretrained model achieved pass@1 of 3/164 (1.8%), and after fine-tuning 9/164 (5.5%).
The author explicitly notes limitations in the HumanEval results: data contamination was not checked, the pretraining and fine-tuning evaluation settings differ (not a controlled measurement), and the saved fine-tuning per-question results file contains NUL bytes and is incomplete.
In the HumanEval pass@1 comparison, phi-1-small (350M parameters) reached 45%, while Keera fine-tuned (97.5M parameters) reached 5.5%.
All Facts (10)
A developer built a 97.5M-parameter Python code model called Keera from scratch, with a total training cost of $1.40.
50%UnverifiedKeera's architecture is based on GPT-2 small: 12 Transformer blocks, hidden dimension 768, and a total parameter count of 97.5M.
50%UnverifiedKeera uses a custom tokenizer with a vocabulary size of 16,384, trained on Python code, at an average of 3.9 characters/token.
50%UnverifiedKeera uses RoPE positional encoding, RMSNorm, weight-tied embeddings, and a 1,024-token context.
50%UnverifiedKeera was pretrained on the 2B-token codeparrot-clean dataset, following the Chinchilla rule of about 20 tokens/parameter.
50%UnverifiedRenting an RTX 5090 on Vast.ai ($0.584 per hour), using bf16 and torch.compile, Keera's pretraining throughput was about 234,000 tokens/second, the 2B-token training took 2 hours 23 minutes, and the validation loss was 1.501.
50%UnverifiedKeera was fine-tuned on the self-oss-instruct-sc2-exec-filter-50k dataset, which contains 50k Python instructions and answers, and all answer code was actually run and passed the tests.
50%UnverifiedOn HumanEval, Keera's pretrained model achieved pass@1 of 3/164 (1.8%), and after fine-tuning 9/164 (5.5%).
50%UnverifiedThe author explicitly notes limitations in the HumanEval results: data contamination was not checked, the pretraining and fine-tuning evaluation settings differ (not a controlled measurement), and the saved fine-tuning per-question results file contains NUL bytes and is incomplete.
50%UnverifiedIn the HumanEval pass@1 comparison, phi-1-small (350M parameters) reached 45%, while Keera fine-tuned (97.5M parameters) reached 5.5%.
50%