PSSA: A Non-Transformer Language Model Built from Scratch in Rust

Going Against the Grain

At a time when the Transformer architecture dominates virtually every corner of the large language model landscape, an open-source project called PSSA has chosen a different path: implemented entirely in Rust from scratch, and deliberately avoiding Transformers altogether. The project recently surfaced on Hacker News and, while the discussion was modest (17 upvotes and 3 comments), the technical direction it represents is worth a closer look.

PSSA's core proposition rests on two pillars — a non-Transformer architecture and a hand-written Rust implementation from the ground up. Neither choice follows the crowd, and both signal an author intent on exploring possibilities outside the established paradigm.

Why Avoid Transformers?

The Transformer architecture, powered by the Self-Attention mechanism, has become the standard foundation for everything from GPT to a wide range of open-source large models, thanks to its parallelization capabilities and ability to model long-range dependencies. But this architecture comes with real costs:

• Attention computation scales quadratically with sequence length, making long-text processing expensive;
• Model parameter counts and training resource requirements keep ballooning, leaving little room for grassroots innovation at the architecture level;
• Architectural homogeneity has funneled a vast amount of research onto a single technical track.

The significance of non-Transformer models like PSSA lies in exploring whether lighter, more efficient, or structurally simpler alternatives exist. In recent years, the research community has seen a wave of attempts — state space models (such as Mamba), linear attention, and others — all trying to break Transformer's near-total dominance. PSSA can be seen as an independent contribution to this wave of exploration.

State space models (SSMs) represent one of the most closely watched alternatives to Transformers today. SSM architectures like Mamba draw on classical control-theory representations, compressing historical sequence information into implicit state vectors rather than explicitly computing pairwise interactions across all positions as attention does. This allows SSMs to achieve linear complexity — O(n) — when processing long sequences, a substantial improvement over standard Self-Attention's O(n²). Linear attention takes a different approach, using kernel function approximations to decouple the attention matrix computation and similarly reach linear complexity. The shared motivation behind all these efforts is to preserve sequence modeling capability while breaking through the quadratic complexity bottleneck, making language models more practical for long-context and resource-constrained scenarios. PSSA's choice of a non-Transformer architecture suggests it may share a design philosophy with this class of methods, though the specifics of its architecture would need to be read from the source code directly.

What Does Choosing Rust Signify?

The vast majority of machine learning projects are built on the Python ecosystem, relying on mature frameworks like PyTorch and TensorFlow. PSSA's decision to write everything in Rust from scratch reflects genuine engineering ambition.

Rust is a systems programming language that delivers performance comparable to C/C++ while providing memory safety guarantees through its ownership model. Applying it to language model implementation offers several potential benefits:

• Controlled performance: No reliance on a heavy Python runtime or third-party frameworks, giving the author full visibility into low-level computation paths;
• Deployment-friendly: Rust compiles to standalone binaries, making it straightforward to run in resource-constrained environments;
• Educational value: Building from scratch forces the author to understand every layer of the model in detail — an invaluable reference for anyone learning the fundamentals.

The trade-offs are equally apparent. Rust's machine learning ecosystem is far less mature than Python's, with no plug-and-play high-level libraries, meaning the author must implement a substantial amount of infrastructure by hand. This underscores what "from scratch" actually means in practice.

Rust's ownership model is the core design that sets it apart from C/C++. The compiler statically tracks the lifetime of every piece of memory at compile time, enforcing guarantees against dangling pointers, data races, and other classic memory safety bugs — without the runtime overhead of a garbage collector. This is particularly relevant for machine learning inference engines: tensor operations involve constant allocation and deallocation of contiguous memory blocks, and Rust can provide stronger safety guarantees than Python without sacrificing performance. While Rust's ML ecosystem is still maturing, frameworks like candle (from HuggingFace) and burn are already exploring this space and demonstrating that neural network inference in Rust is engineering-viable. Building from scratch means the author cannot rely on any of these frameworks' ready-made operators — they must implement matrix multiplication, activation functions, forward passes, and other foundational modules themselves. That is where the real technical weight of "from scratch" lies.

How to Assess an Independent Project Like This

It's worth keeping perspective: judging by the limited discussion on Hacker News, PSSA currently looks more like a technical exploration by an individual or small team than a production-grade system capable of competing with mainstream large models. Its value is most evident across a few dimensions:

First, teaching and research reference. For developers who want a deep understanding of how language models work internally, a Rust implementation that doesn't rely on black-box frameworks and prioritizes code readability is an excellent learning resource.

Second, an architecture experimentation platform. The non-Transformer design provides a clean canvas for trying out novel sequence modeling approaches, free from the constraints of patching existing frameworks.

Third, a demonstration of engineering possibilities. It proves that implementing a complete language model in a systems language is feasible and offers a useful perspective for scenarios where performance and deployment efficiency matter.

Worth Watching

The direction PSSA represents aligns with a growing sentiment in the AI community: a desire to find more open, more controllable, and more understandable technical paths outside the big-player arms race in large language models. Regardless of how far this specific project ultimately goes, the practice itself — building from scratch with a non-mainstream language and a non-mainstream architecture — adds meaningful diversity to the broader ecosystem.

For developers with an interest in low-level technology, PSSA is an open-source effort worth following and worth digging into hands-on. It's a reminder that there is no single way to implement a language model, and that technical progress so often originates from exactly these kinds of off-the-beaten-path explorations.