Training a Language Model in Pure Rust: The Framework Pitfalls Behind a $164 Experiment

An independent researcher did something almost no one has attempted before: completely abandoning PyTorch and Python, training a language model end-to-end from scratch using only Rust. The entire training run cost just $164 in rented GPU time. But the real value of this experiment report isn't in the flashy achievement — it's a sober, quantifiable "failure log" of the Rust machine learning ecosystem.

A Solitary Pure-Rust Training Experiment

The author completed this work alone — no team, no PyTorch, and no Python anywhere in the training pipeline. The resulting model has approximately 0.4 billion parameters, with Bangla as the primary target language. Training used roughly 2 billion tokens of corpus data, running for 54.6 hours on a single rented H100.

The author is explicit that this is being reported as an "accomplishment," not a "recommendation." In other words, he does not suggest others train large models in Rust right now. The more meaningful contribution of this report is its systematic documentation of the shortcomings exposed by the two mainstream Rust machine learning frameworks — Candle and Burn — when used as training backends (as opposed to inference backends) in 2026. There is virtually no prior public stress-testing record for this use case.

Silent Failures: Framework Bugs Invisible to the Naked Eye

The most alarming finding is that nearly all of these framework bugs are "silent." They don't throw errors, don't crash, and can even pass routine loss curve checks — the loss appears to be declining normally, while something has already gone deeply wrong inside the model.

The author documented five bugs in Candle and three in Burn. Several are particularly severe:

• Candle's fused kernels silently produce zero gradients — meaning certain parameters receive no updates during training, while everything appears normal on the surface.
• Burn's backward pass achieves only about 3% of theoretical GPU throughput, meaning the vast majority of compute is wasted.
• Burn's kernel fusion path segfaults mid-training at the scale of billions of parameters.

What these issues share is that not one of them announces its own existence. They lurk beneath normal training metrics, and without targeted validation, could easily go completely unnoticed — ultimately yielding a model that quietly failed to train.

Candle and Burn are the two most mainstream machine learning frameworks in the Rust ecosystem. Candle is developed primarily by HuggingFace, positioned as a lightweight, low-level tensor computation library with a minimalist design philosophy. Burn is a community-driven project aiming to provide higher-level abstractions similar to the PyTorch experience, with support for multiple backends (including CUDA, WebGPU, etc.). Both have seen some production use in inference scenarios, but training — especially pretraining tasks requiring precise backpropagation and large-scale parameter updates — demands far higher correctness from a framework than inference does, with almost no prior public stress-test records.

The Gradient-Flow Arbiter: A Reusable Validation Discipline

The author's ability to catch six silent failures in this minefield of a codebase came down to a rigorous validation discipline, centered on what he calls the gradient-flow arbiter test.

The logic of this test is straightforward yet highly effective: run a single forward/backward pass, then assert that every trainable parameter has received a finite, non-zero gradient. If any parameter receives a zero gradient or an illegal value (such as NaN or Inf), the test fails immediately.

The value of this approach lies in its universality — it doesn't depend on any specific framework and can be applied to any deep learning training system as a basic health-check safeguard. For any team building a custom training pipeline, this is an engineering practice worth adopting: don't just watch the loss curve; directly verify that gradients are actually flowing through the entire network.

The necessity of gradient flow validation stems from a structural vulnerability in deep learning training: a declining loss function does not mean all parameters are being correctly optimized. Vanishing gradients, accidentally frozen parameters, broken computation graphs, or kernel implementation bugs at the framework level can all cause gradients to be zero for certain parameters — while the model's overall loss continues to slowly decrease due to updates from other parameters, making everything look fine from the outside. In the PyTorch ecosystem, these issues are typically caught by a mature community, extensive known-case documentation, and thoroughly tested autograd. In a less mature framework, developers must build this defense themselves. Making gradient checking a standard assertion in the training pipeline — rather than a temporary debugging tool — is a defensive engineering habit with practical value in any custom training framework or non-mainstream backend.

Bangla Tokenization's "Fertility Trap"

Beyond framework-level issues, the author also revealed a hidden pitfall at the intersection of linguistics and engineering: the tokenizer-fertility trap inherent to the Bengali script.

Using naive byte-level tokenization initially compressed Bangla to an average of roughly 1.4 characters per token, while English achieved about 3.9 characters per token. This disparity had serious consequences: it silently inverted the linguistic balance of the corpus — despite the training data being primarily Bangla, the low tokenization efficiency meant that the effective amount of Bangla information fed to the model was severely diluted.

After fixing this, Bangla tokenization efficiency improved to approximately 4.1 characters per token, restoring the language balance. This case reminds us that in multilingual model training, a tokenizer's handling efficiency for non-Latin scripts directly and covertly affects training outcomes.

Tokenizer fertility refers to the average number of tokens a piece of text is encoded into — how many tokens correspond to each word or character. For Latin-alphabet languages like English, mature BPE (Byte Pair Encoding) tokenizers can merge common words into single tokens efficiently. For non-Latin scripts like Bengali, Chinese, or Arabic, if the tokenizer vocabulary is primarily built from English-language corpora, large numbers of characters get downgraded to byte-level representations, causing the same semantic content to consume several times the token budget. In mixed-language pretraining, this efficiency gap not only wastes sequence length but also effectively applies hidden weights to different languages — the language with lower efficiency has less effective information per training step, and its semantic signal is systematically diluted. This is a problem repeatedly emphasized in multilingual large model research yet still commonly overlooked in engineering practice.

Training Results and Final Conclusions

Looking at the results, this carefully budget-controlled small model did learn something real. It showed strong signals on Bangla language modeling: a negative log-likelihood (NLL) per token of 0.93, compared to 12.60 for a randomly initialized twin model with the same architecture — a striking difference.

Meanwhile, the model performed near random chance on English commonsense multiple-choice questions — entirely as expected, given that this was a deliberately small, Bangla-focused budget training run (approximately 2 billion tokens). Resources were intentionally concentrated on a single language.

The author considers this likely to be one of the earliest known records of end-to-end language model pretraining in pure Rust. But after the experiment concluded, he made a pragmatic decision: moving training back to PyTorch, keeping Rust only for on-device inference serving.

His final assessment is clear: for now, Rust is not a competitive environment for language model training, but it may be a strong environment for model deployment and serving. This division of labor — "train with PyTorch, serve with Rust" — is perhaps the most realistic technical combination for the current moment.