15-Year-Old Developer Builds a Hardware-Aware SLM Trainer from Scratch: Python and Rust in Practice

A 15-year-old developer chose not to use high-level wrappers or fine-tune an existing large model, but instead decided to spend several months building an end-to-end training pipeline from scratch to truly understand how Transformers work. This project, called "Hardware-Aware SLM Trainer," combines Python and Rust in an attempt to train small language models (SLMs) on consumer-grade hardware. The project is open-sourced on GitHub, and the author has also shared current progress and challenges encountered.

Project Core: Hardware-Aware Model Size Decisions

The most distinctive aspect of this trainer is its "hardware awareness." It automatically detects your hardware configuration (NVIDIA, AMD, or CPU), then runs a quick matrix multiplication benchmark to measure the hardware's raw TFLOPS throughput. This data is fed into a helper program written in Rust.

The Rust engine uses the compute throughput and memory budget to calculate how large a model the local GPU can train within specific time and VRAM constraints (supporting up to 100 million parameters) without running out of VRAM. This approach addresses a common pain point for individual developers: not knowing how large a model their hardware can handle, often relying on trial and error and wasting significant time.

The author documents structural trade-offs and decisions in the codebase, making it excellent learning material for anyone wanting to understand how training loops operate on constrained hardware.

TFLOPS (trillions of floating-point operations per second) is a common metric for measuring hardware compute throughput. Consumer GPUs' rated TFLOPS typically refers to FP32 or FP16 peak values, but in actual training, effective compute is often significantly lower than peak due to memory bandwidth, kernel launch overhead, matrix shapes, and other factors. Therefore, estimating model size based solely on rated TFLOPS can easily overestimate hardware capability; in practice, it's common to use benchmark-measured values and leave a margin.

VRAM budget is another key constraint. During training, VRAM usage includes not only model parameters but also gradients, optimizer states (such as Adam's momentum and variance), activations, and temporary buffers. For a model with P parameters, mixed-precision training roughly requires tens of times the parameter count in bytes. As a rough estimate, you can plan for 16-20 bytes per parameter (including parameters, gradients, and optimizer states) plus activation overhead—this is also why the hardware-aware module needs to make decisions based on memory budget rather than just compute.

Data Pipeline: Extracting Papers from arXiv and Protecting LaTeX

The author is currently working on the data ingestion pipeline. The specific task is to extract raw machine learning papers from arXiv, clean them, and ensure that LaTeX code blocks are not broken during tokenization.

This detail is crucial. If formulas and code blocks in academic papers are incorrectly tokenized, it introduces noise that affects model learning. The author's deliberate handling of this shows serious consideration for data quality. Although the project is still in its early stages, this attention to data details is commendable.

Tokenization is the process of splitting raw text into subword units (tokens) and mapping them to integer IDs—the first step in language model training. The common BPE (Byte Pair Encoding) algorithm starts from characters, repeatedly merges high-frequency adjacent pairs, and eventually forms a vocabulary. For LaTeX in academic papers, the problem is that backslashes, curly braces, and mathematical symbols get split into fragments, breaking the semantic structure of formulas and causing the model to learn incorrect patterns. Common approaches to protecting LaTeX code blocks include: replacing formula blocks with placeholders before tokenization, preserving formulas as whole units, or filtering out fragments that cannot be parsed correctly during the cleaning stage. These processing steps directly affect data quality and final model performance.

Tech Stack Choice: Division of Labor Between Python and Rust

The project uses a hybrid of Python and Rust. Python handles the high-level training pipeline and data processing, while Rust is used for performance-sensitive hardware benchmarking and model size calculation. This division of labor is increasingly common in machine learning engineering: Python has a rich ecosystem suitable for rapid iteration; Rust offers good performance suitable for low-level optimization.

The author did not disclose more implementation details, but from the description, the Rust part is a standalone helper binary that interacts with Python through pipes. This design keeps modules clean and facilitates separate testing.

Hybrid Python-Rust development is gaining popularity in machine learning infrastructure, with typical patterns being using PyO3 or maturin to compile Rust into Python-callable extension modules, or interacting via subprocesses/pipes. Rust's advantages include no GC, memory safety, and high performance, making it suitable for compute-intensive tasks like benchmarking, data preprocessing, and tokenizers; Python relies on ecosystems like PyTorch and NumPy for model definition, training loop scheduling, and experiment management. The "standalone helper binary interacting with Python through pipes" mentioned in the article is a loosely coupled approach, with the advantage of clear boundaries and independent testability, at the cost of serialization overhead in inter-process communication, making it suitable for low-frequency calls (such as one-time hardware evaluation).

Community Feedback and Learning Advice

The author posted on Reddit seeking experience and advice on managing training loops on constrained consumer hardware (such as laptop GPUs). For a 15-year-old developer, this from-scratch learning approach is extremely valuable. It not only exercises programming skills but also provides deep understanding of Transformer internals.

If you're interested in training small language models or want to learn about hardware-aware model size estimation, this project provides an excellent starting point. The author's open-source spirit and documentation habits are also worth emulating.

Key Takeaways

• The project builds an end-to-end training pipeline from scratch rather than relying on high-level wrappers, aiming to deeply understand Transformers.
• The hardware-aware module uses benchmarking and a Rust engine to automatically calculate trainable model size (up to 100 million parameters), avoiding VRAM overflow.
• The data pipeline extracts papers from arXiv, with special attention to protecting LaTeX code blocks during tokenization.
• It uses a hybrid Python and Rust tech stack, balancing development efficiency and performance.
• The author is a 15-year-old developer; the project is open-source and seeks community feedback, embodying the spirit of learning and sharing.

One-line Summary

A 15-year-old developer builds a hardware-aware small language model trainer from scratch using Python and Rust, automatically estimating trainable size and protecting LaTeX data.