Janas: Streaming Experts from SSD to Push LLMs Beyond Memory Limits

What Do You Do When the Model Is Bigger Than Your Memory?

LLM parameter counts keep ballooning, while the memory capacity of consumer-grade hardware struggles to keep up. This raises a straightforward question: if a model file is larger than available memory, can it still be run?

A project called Janas recently surfaced on Hacker News, and its core claim is simple yet compelling — run LLMs that exceed memory capacity by streaming experts from SSD.

The key lies in the word "experts." This points to the Mixture-of-Experts (MoE) architecture: a model composed of multiple expert sub-networks, only a small subset of which is activated per inference. Since most experts don't participate in a single forward pass, there's no need to cram them all into memory — just read them from SSD on demand.

Janas turns this observation into an engineering implementation: expert weights are stored on SSD and dynamically streamed in during inference. Memory only needs to hold the currently activated experts and the necessary runtime state, while the total model size can far exceed the RAM ceiling.

Mixture-of-Experts (MoE) architecture is a sparsely activated design that splits a single large model into multiple "expert" sub-networks. Each expert is essentially a set of independent parameters (typically the FFN weights in a feed-forward layer), and a lightweight "router" network decides which few experts each input token gets assigned to based on its features. In a typical configuration like Mixtral 8x7B, each layer has 8 experts, but each token is routed to only 2 of them, so the activated parameter count per pass is roughly one-third of the total (plus shared weights). The benefit: greater parameter capacity for less compute, since different experts can each learn different knowledge domains or language patterns. The cost: harder training, more complex memory management, and the need to handle dynamic scheduling of multiple experts at inference time. It's precisely this "large total, small activation" property that provides the theoretical premise for streaming schemes like Janas — idle expert parameters can be pulled in on demand from slower storage rather than residing in memory permanently.

Why MoE Is Especially Suited to Streaming

Traditional dense models use all parameters for every inference. No matter how you optimize, the full weights must reside in memory or VRAM — otherwise every computation step triggers a disk read, and the speed becomes unacceptable.

MoE changes this premise. In a typical sparse MoE, each token is routed to only a handful of experts, leaving the rest idle. The total parameter count may be large, but the per-activation parameter count is far smaller. This "large total, small activation" property is naturally suited to keeping cold data on slow but high-capacity storage media.

This is where Janas's engineering value lies: it decouples "what fits" from "what's needed." SSD handles what fits; memory handles what's needed; streaming scheduling bridges the two. For users with large-capacity NVMe SSDs but limited RAM, this path makes models that previously couldn't run now runnable.

The Cost and Trade-offs of Streaming

Streaming experts from SSD isn't free. SSDs deliver much higher sequential read bandwidth than random read bandwidth, yet expert scheduling often exhibits a degree of randomness — different tokens may hit different experts, fragmenting the read pattern. Actual performance therefore depends heavily on several factors:

• SSD random read capability: NVMe has a clear advantage over SATA
• Expert hit patterns: whether routing is concentrated and whether there's cache reuse
• Prefetch strategy: whether the next set of needed experts can be predicted in advance
• Memory cache size: how many experts can be cached to reduce repeated reads

These trade-offs determine that Janas is better suited for "getting it to run" than "getting it to run fast." For researchers and enthusiasts with severely limited memory who nonetheless want to experiment with large models, this is a reasonable compromise.

Sequential vs. random reads on SSD differ significantly in performance. NVMe SSDs typically achieve 3–7 GB/s sequential read bandwidth, and while 4KB-granularity random read IOPS are high (high-end drives can reach millions), per-read latency remains in the microsecond range, and effective throughput drops sharply when read patterns fragment. SATA SSDs are limited by the 6 Gbps interface, delivering only about 550 MB/s sequential bandwidth, with an even wider gap in random performance. When streaming MoE experts, each token may hit a different expert, producing random weight-read patterns — if expert weight blocks are large (an expert may be hundreds of MB), read granularity can approach sequential, keeping performance loss relatively contained; but if scheduling is extremely fragmented, effective bandwidth may be only a fraction of sequential reads. This is why schemes like Janas depend heavily on NVMe: on SATA drives, random read latency is too high, and streaming inference speed plummets to unusable levels. Prefetch strategies and expert caching exist precisely to mitigate this randomness — by predicting which experts will activate next and reading them in advance, random access is converted into controllable asynchronous sequential reads.

Implications for the Local Deployment Landscape

Janas represents an engineering direction worth watching: trading storage for memory. With memory and VRAM prices staying high, using large-capacity SSDs as "secondary storage" for model parameters offers another path to running very large models locally.

It doesn't conflict with quantization, distillation, KV Cache optimization, and similar techniques — in fact, they stack. Quantization shrinks each expert's size, streaming solves the total-volume problem, and together they can further lower the hardware barrier.

Of course, this direction has clear boundaries. SSD lifespan, read latency, and throughput ceilings are all physical constraints; streaming can't bring inference speed close to an all-in-memory solution. It solves "can it run," not "how fast does it run."

Summary

Janas's idea isn't complicated, but it hits a real pain point: models are growing faster than hardware memory. By combining MoE's sparse activation with SSD's high capacity, it makes models that exceed memory at least possible to run.

For developers focused on local LLM deployment and MoE inference optimization, this project offers a sample worth studying. Its value isn't in performance numbers but in demonstrating an overlooked engineering path — when memory isn't enough, the disk can be part of the answer.

One-line Summary

Janas leverages MoE sparse activation to stream experts from SSD, making LLMs larger than memory runnable.

Paragraph Summary

This article introduces Janas, a project on Hacker News that streams Mixture-of-Experts (MoE) sub-networks from SSD so total model size can far exceed available memory. Since MoE activates only a few experts per inference, the rest need not reside in memory and can be read on demand. The scheme decouples "what fits" from "what's needed": SSD provides capacity, memory handles runtime state. But streaming comes with performance costs depending on SSD random read capability, expert hit patterns, prefetch strategy, and cache size, making it better suited for "getting it to run" than "getting it to run fast." The idea complements quantization and distillation to further lower the local deployment barrier — solving whether it can run, not how fast.