Training a 97.5M-Parameter Python Model from Scratch for Just $1.40

Title

Training a 97.5M-Parameter Python Model from Scratch for Just $1.40

Meta Description

A developer trained Keera, a 97.5M-parameter Python code model, from scratch with only $1.40 in GPU rental costs, completing pretraining in 2 hours 23 minutes. A full practical log covering the GPT-2-style architecture, custom tokenizer, fine-tuning pipeline, and HumanEval benchmark.

Article Content

How much does it cost to train a language model? One developer answered this with practice: $1.40. He built Keera, a 97.5M-parameter Python code model, from scratch, walking through the entire pipeline of tokenizer, pretraining, fine-tuning, and benchmarking, all while keeping costs under the price of a cup of coffee.

Model Architecture: A Modern Replica of GPT-2 small

Keera's skeleton is almost a carbon copy of GPT-2 small: 12 Transformer blocks, hidden dimension 768, 97.5M total parameters. Each block performs two operations—first attention (tokens look at each other), then MLP (each token processed independently), with residual connections in between letting information flow straight through.

x = x + Attention(RMSNorm(x))
x = x + MLP(RMSNorm(x))

In terms of parameter distribution, the 12 blocks account for 84.9M (87%), while the embedding layer is kept to 12.6M (13%) thanks to shared output weights. The author's reasoning is clear: replicate a known workable approach at this scale, and only swap in newer components when they're cheaper or better.

Several key design decisions are worth noting:

• Custom tokenizer, 16,384 vocabulary: Trained on Python code, preserving indentation as distinct tokens, averaging 3.9 characters/token. Compared to GPT-2's 50k vocabulary, embedding layer parameters drop from 38.6M to 12.6M, leaving more parameter budget for the Transformer blocks.
• RoPE positional encoding: Encodes relative positions by rotating query and key, adding no learned parameters.
• RMSNorm: Normalizes before each sublayer without mean subtraction or bias, for stable training.
• Weight-shared embeddings: Input and output layers share the same matrix, saving 12.6M parameters.
• 1,024-token context: Most Python functions fit; attention cost grows quadratically with length, so keeping it modest is wise.
• 2B tokens of training data: Following the Chinchilla rule of roughly 20 tokens/parameter, 20 × 97.5M ≈ 1.95B, a near-exact match.

Residual connections are the key structure enabling stable training of deep Transformers: they add the input of a sublayer (attention or MLP) directly to its output, forming x = x + f(x). This allows gradients to flow back losslessly through the identity path, mitigating vanishing gradients in deep networks; meanwhile each sublayer only needs to learn an "incremental correction" to its input, simplifying the optimization target. The smooth convergence of Keera's 12 stacked layers owes much to the combination of residual connections and RMSNorm.

Pretraining: 2 Hours 23 Minutes, $1.40

The data is codeparrot-clean—a deduplicated collection of Python files from GitHub, tokenized into a 4GB file with EOF markers inserted between files. During training, 1,024-token windows are randomly sampled.

The author first ran a 100M-token test on a MacBook, taking 9 hours 24 minutes. At that rate, 2B tokens would require about 190 hours, so he turned to renting a GPU.

He rented an RTX 5090 on Vast.ai at $0.584 per hour. With bf16 and torch.compile, throughput reached about 234,000 tokens/second—77 times the MacBook training setup. The final 2B-token pretraining took 2 hours 23 minutes, with validation loss of 1.501 (perplexity around 4.5).

Compared to the loss of 2.125 when the MacBook test run stopped, this is a clear improvement. But the author also candidly notes that the batch sizes and schedules differ, so it's not a controlled batch-size comparison.

However, the pretrained model writes code that "looks right but is often wrong." For example, asked to make is_even(n) check for even numbers, it returns if n > 0: return False—the shape is right, the logic is completely wrong.

Practical Lessons from Renting a GPU

The author ran into a few pitfalls on Vast.ai worth recording:

• Run a benchmark before training: 30 steps is enough to measure tokens/second, with the cost estimate formula tokens ÷ tok/s ÷ 3600 × hourly price. In this case: 2e9 ÷ 236,000 ÷ 3600 × $0.584 ≈ $1.40.
• Choose on-demand instances: He first tried two cheaper interruptible machines, but they were preempted and paused by higher bidders before training even started.
• Sync code with git archive: Excludes untracked files and caches, but note that committed files (including secrets) get packaged too.
• Destroy instances immediately after use: Stopped instances still incur disk charges.

bf16 (Brain Floating Point 16) is a 16-bit floating-point format that, compared to FP16, retains the same 8-bit exponent range as FP32 with only lower mantissa precision. This means gradient underflow or overflow is less likely during training, and it can typically be used directly without loss scaling. On GPUs supporting bf16 such as the RTX 5090, matrix multiplication throughput is far higher than FP32 while memory usage is halved—making it the standard precision choice for large-model pretraining.

Fine-tuning: Teaching the Model to Answer Questions

The pretrained model only continues text; ask it a question and it may keep writing more questions. The author fine-tuned on the self-oss-instruct-sc2-exec-filter-50k dataset—50k Python instructions and answers, where each answer's code was actually run and passed tests.

The data is wrapped in a simple chat format; the model reads the whole segment but only computes loss on the answer portion:

<|user|> Write a function that reverses a string. <|end|> <|assistant|>
Here's how … def reverse(s): return s[::-1] … <|end|>

The 49,549 training samples packed into 16,121 rows (1,025 tokens per row, about 96% fill rate), ran for about three epochs on the MacBook, taking 4 hours 40 minutes.

Validation loss essentially plateaued at 0.81 by the third epoch, while training loss kept dropping to 0.47, suggesting diminishing returns and potential overfitting. The author says he'll compare earlier checkpoints next time, but hasn't yet measured whether two epochs reach the same HumanEval score.

The model's grasp of answer format was nearly immediate—by step 100 it could write a short intro, a code block, a one-line explanation, then stop. But logical correctness came much more slowly. The author self-tested 10 prompts: 1 correct, 3 close, 6 wrong.

Instruction tuning is the process of continuing to train a model after pretraining on "instruction-answer" paired data, shifting it from mere text continuation to following user intent. The key is that loss is computed only on the answer portion (after <|assistant|>), with the question portion masked, so the model learns the conditional distribution of "generate answer given question" rather than restating the question. Data quality matters more than quantity—if answers are execution-verified, this significantly reduces the model learning wrong patterns.

HumanEval Performance: 5.5%, with Caveats

HumanEval has 164 Python problems, with unit tests hidden from the model. The author used greedy decoding to generate one answer per problem and reported pass@1. Each answer runs in a separate process with a temporary working directory and 10-second timeout. The scorer was validated in both directions: all 164 official answers pass, all empty answers fail.

Results: pretrained model 3/164 (1.8%), fine-tuned 9/164 (5.5%).

But the author explicitly flags the limitations of these results:

• No data contamination check: Whether training data overlaps with HumanEval has not been verified.
• Different evaluation setups before and after: Pretraining used 300-token continuation, fine-tuning used chat instructions plus 512 tokens, extracting the first code block for testing. The change in prompt format and token budget means this is not a controlled measurement of fine-tuning effects.
• Incomplete results file: The saved per-problem fine-tuning results file contains NUL bytes and needs recovery or a rerun to be considered fully reproducible.

Comparison with public models (different evaluation setups, not a ranking on the same benchmark):

Model	Parameters	Training Data	HumanEval pass@1
GPT-Neo 125M	125M	General web text	0.8%
Keera pretrained	97.5M	2B tokens Python	1.8%
codeparrot-small	110M	Same dataset	3.8%
GPT-Neo 1.3B	1.3B	General web text	4.8%
Keera fine-tuned (preliminary)	97.5M	+50k instructions	5.5%
Codex 85M	85M	GitHub code	8.2%
phi-1-small	350M	"Textbook-quality" data	45%

phi-1-small's 45% is particularly striking—it used filtered web data, synthetic textbooks, and coding exercises. The author believes this suggests data quality deserves significant investment, but phi-1-small has more parameters and a different training budget, so it can't be attributed to data alone.

pass@1 is the standard evaluation metric in code generation: generate only one answer per problem, count it as 1 if it passes all unit tests and 0 otherwise, then average the pass rate. It measures the probability of "getting it right on the first try," stricter than pass@10 (at least one of 10 answers passes). HumanEval contains 164 hand-written Python problems, each with hidden test cases the model cannot see, so it reasonably reflects generalization—but watch out for data contamination from overlap between training data and the test set.

Cost Breakdown

Step	Location	Time	Cost
Tokenizer + data	MacBook	minutes	–
100M-token test	MacBook M3 Pro	9 hours 24 min	–
2B-token pretraining	RTX 5090, Vast.ai	2 hours 23 min	$1.40
Fine-tuning	MacBook	4 hours 40 min	–
HumanEval	MacBook	about 20 min	–

The $1.40 covers only the rental cost of the main pretraining run, excluding environment setup, data preparation, storage, transfer, earlier rental attempts, and laptop hardware depreciation.

Key Takeaways

The author summarized several lessons worth remembering:

Loss and task accuracy are two different things. 20x more training tokens dropped loss from 2.13 to 1.50, but both models solved only 2 of the author's 9 informal test tasks—and the successful ones differed.

Watch the validation curve during fine-tuning. Validation loss improved little by the third epoch; earlier checkpoints are worth benchmarking too.

Limiting GPU memory on Mac helps. Setting PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.6 (with a low watermark of 0.5) resolved swapping and GPU stuttering on the 18GB M3 Pro.

Test on small models first. A CPU-sized miniature caught most of the bugs before any money was spent.

Next Steps

• Better data: Educational Python, drawing on phi-1's lessons.
• More reproducible numbers: Recover the full HumanEval output, check for contamination, compare checkpoints under matched evaluation setups, and add MBPP and pass@10.
• Fill-in-the-Middle (FIM) training: Let the model auto-complete inside editors; special tokens are already in the tokenizer.
• Scaling up: Next target is 350M, then possibly 1B. At 20 tokens/parameter, a 1B model needs about 20B tokens; benchmark first, then estimate rental costs.
• Longer context: Extend RoPE from 1,024 to 4,096 tokens.
• Preference tuning (DPO): Layered on top of fine-tuning.

Worth noting: the author used AI extensively during the build—Claude Opus 5.5 (in Claude Code) wrote most of the code and tests, and Codex helped review implementations and check results. But the decisions, experiments, GPU rentals, and result validation were all done by the author himself.

The value of this project isn't in the 5.5% number itself—it's still far from a practical coding assistant. The value is in walking the full pipeline at extremely low cost, making every step understandable and adjustable. For anyone wanting to deeply understand the internals of language models, this is more convincing than reading ten papers.

One-line Summary

A developer trained a 97.5M-parameter Python code model from scratch with $1.40 in GPU rental, covering the full pipeline.

Paragraph Summary

A developer built Keera, a 97.5M-parameter Python code model, from scratch, walking the entire pipeline of tokenizer, pretraining, fine-tuning, and HumanEval benchmarking. The model uses a GPT-2 small architecture with modern components like RoPE, RMSNorm, and weight sharing, and a custom 16,384-vocabulary tokenizer that compresses the embedding layer to 12.6M parameters. Pretraining 2B tokens on a rented RTX 5090 on Vast.ai took only 2 hours 23 minutes at a cost of $1.40. After fine-tuning on 50k instruction samples, HumanEval pass@1 improved from 1.8% to 5.5%, though the author explicitly notes differing evaluation setups and contamination risks. The article emphasizes that lower loss doesn't equal higher task accuracy, and shares practical lessons like running benchmarks to estimate costs before renting a GPU.