Janas Engine Runs 48GB MoE Model on a 32GB Laptop at 37 tok/s, Beating llama.cpp

Title

Janas Engine Runs 48GB MoE Model on a 32GB Laptop at 37 tok/s, Beating llama.cpp

Meta Description

Janas is a MoE inference engine written in C that streams expert weights from SSD via O_DIRECT, letting a 32GB laptop run a 48GB model at 37 tok/s, beating llama.cpp across 13 model benchmarks.

Article Content

For local LLM enthusiasts, this scenario is all too familiar: you finally find a MoE model you love, then see the file is 48GB, glance at your 32GB of RAM, and quietly close the tab. Even if you grit your teeth and load it, all you get is a glacial 3 tokens per second and an SSD activity light that never stops.

The Janas project, which hit the Hacker News front page this week, claims to break through that wall. It's a language model inference engine written in C, and the core numbers from author Maurizio Cammalleri are eye-catching: running a 48.4GB Qwen3-Next-80B on a 32GB laptop at 37 tok/s. On the same machine with the same model, llama.cpp manages only 2.8 tok/s.

First, a caveat: all figures come from the project's public benchmarks, README, and Hacker News discussion threads. These are the author's own measurements, not independent third-party verification, so read with that in mind.

Why Models That Exceed RAM Crawl Like a Slideshow

A natural question: doesn't the operating system have built-in memory mapping? The model file sits on the SSD, and the system can page it in on demand — that's exactly how llama.cpp uses mmap.

The problem arises once the working set exceeds physical memory. Memory-mapped files don't know which parts of the model matter, and every page fault has to go through the kernel's page cache path. When memory runs out, the kernel starts evicting the very pages you're about to need. Researchers call this a "page fault storm": once a model reaches roughly twice the size of memory, the mmap approach collapses, with the OS blindly shuffling pages while inference grinds to a halt on disk I/O.

The MoE architecture amplifies this problem from a different angle — and also hints at the solution. A MoE model contains multiple independent "expert" feed-forward blocks, and each token generated is routed to only a few of them, typically 8 out of 256. The vast majority of parameters sit idle at any given moment. The real bottleneck is: how do you quickly read the right slice of parameters for every token?

Page Cache is the mechanism by which the operating system uses free physical memory to cache disk file contents. When a program reads a file, the kernel first reads the data into the page cache, then copies it to the program; the next read of the same file can hit the cache directly with no disk I/O. This can massively boost performance when memory is plentiful, but the cache is "opportunistic": as soon as other processes need physical memory, the kernel evicts pages from the page cache. For large models that exceed physical memory, the page cache almost never hits — just-loaded weight pages are promptly evicted, and the next access triggers another disk read, thrashing back and forth.

O_DIRECT is a Linux file-open flag that tells the kernel to bypass the page cache and transfer data straight from disk to user-space buffers. This sacrifices the cache's acceleration but buys two key benefits: first, memory usage is entirely under the program's control, and the kernel won't blindly evict useful pages due to page cache contention; second, it avoids the double-copy overhead of "data copied into page cache, then copied to the program." Janas uses O_DIRECT precisely to read expert weights on demand, keeping only the hot experts it actively caches in memory — thereby sidestepping the system-level page fault storm. The cost is that every read must be block-aligned and depends entirely on the SSD's random read performance.

Janas's Core Design Approach

Janas's design is relatively credible because its principles are simple enough:

Weights stay on disk. The engine never loads the entire model. It places the converted .jns file on an NVMe SSD and reads expert weights with O_DIRECT, completely bypassing the page cache so the OS never thrashes.

Hot experts go into a memory cache. During runtime the engine tracks which experts are called repeatedly and keeps high-frequency experts in a memory cache. On startup it probes local hardware and automatically decides cache size and thread count accordingly.

The GPU handles only the prompt, not the whole model. On machines with integrated GPUs, Janas uses the GPU for prompt processing (the stage that benefits most from parallelism), while the token generation phase streams expert weights from the SSD. You can force pure CPU mode with JANAS_GPU=0.

Draft tokens hide latency. The engine supports speculative decoding: a small draft model guesses the next few tokens, and the large model verifies them in one pass. On Qwen3-Next-80B, this boosts write speed from 24 to 37 tok/s, with text content completely identical — just generated faster.

On the author's test machine (Debian 13, Intel Core Ultra 9 185H, 32GB DDR5, 1TB NVMe at roughly 5.7 GB/s read), Qwen3-Next-80B reads a 1024-token prompt at 62 tok/s and writes at 24 tok/s without drafts. The equally-sized 48.4GB Qwen3-Coder-Next manages 66 read and 22 write.

Speculative Decoding is a technique for accelerating large model generation without changing the output distribution. The principle: use a much smaller "draft model" to quickly generate a candidate sequence of the next several tokens, then have the large model verify in parallel whether these candidate tokens match what it would have generated itself. Because the large model can verify a batch of tokens in a single forward pass, whereas normal generation of the same number of tokens requires multiple forward passes, when the draft hit rate is high the number of large-model calls drops significantly.

The key is that the verification process mathematically guarantees the final output is exactly identical to what the large model would generate alone — rejected candidates are discarded and resampled from the rejection point. So speculative decoding is "lossless" acceleration: it affects only speed, not quality. In Janas's case, the large model's bottleneck is streaming expert weights from SSD for every token, and speculative decoding lets one read verify multiple tokens, directly reducing per-token disk I/O overhead — which is precisely why the article's write speed goes from 24 to 37 tok/s while the text content stays the same.

Model-by-Model Benchmark Comparison

The author claims Janas beats llama.cpp on all 13 compared models, under identical test conditions on the same laptop with Q4_K_M quantization. These are the author's self-measured data, so discount accordingly.

Models exceeding memory are the most compelling scenario. Qwen3-Next-80B-A3B (48.4GB, larger than physical memory) hits 62 tok/s read and 24 tok/s write on Janas, versus 16 and 2.8 for llama.cpp — a write speed gap of about 8.5x. Qwen3-Coder-Next (48.4GB) writes at 22 vs 2.2.

On models that fit in memory, the gap narrows but persists. Qwen3.6-35B-A3B (22.3GB) reads 104 vs 79, writes 19 vs 12; Qwen3-30B-A3B (18.6GB) writes 32 vs 15; Gemma-4-26B-A4B-it (18.3GB) writes 18 vs 12. Even on the tiny Qwen3-0.6B, Janas reads at 939 tok/s vs 670.

One detail worth noting: for very large models, llama.cpp doesn't just slow down — it outright crashes. The standard build runs out of memory as soon as the GPU is enabled, because its CPU backend copies weights for reordering, and the Vulkan backend copies CPU-side weights into buffers the system can't reclaim. That 2.8 tok/s figure required a specially compiled version with extra parameters. Janas runs the same model with no special configuration whatsoever.

Skeptics on Hacker News raised a fair point: llama.cpp has supported mmap-based disk streaming for years, and projects like Strata and flash-moe are doing similar things. If these numbers hold up on other machines, Janas's difference lies in actively probing hardware and managing which experts reside in memory, rather than leaving paging decisions to the kernel. The author also explicitly states that performance on untested hardware is unknown, and the janas-bench report is designed precisely to answer that question.

Q4_K_M is the name of a quantization format widely used in the llama.cpp community. GGUF model files support multiple quantization levels; the letter "Q" stands for Quantization, and the number "4" means weights are compressed to roughly 4 bits. The "K" suffix indicates the k-quant mixed quantization scheme — using different quantization precision for different layers or tensors to better balance compression ratio and accuracy; "M" is the Medium tier within that scheme. Q4_K_M has become the default choice for locally deployed MoE models because it achieves a good compromise between the size advantage of 4-bit quantization and output quality close to the original model.

Understanding this matters for interpreting the comparisons here: both Janas and llama.cpp benchmarks use Q4_K_M, meaning both have identical model file sizes and numerical precision, and the difference comes purely from the inference engine's scheduling strategy rather than the quantization itself. Additionally, quantized weights are stored on disk in blocks, which explains why O_DIRECT reads and expert cache management have such a large performance impact — the engine needs to efficiently move these compressed blocks from SSD to memory and decompress them for use.

Getting Started Guide and Caveats

Quick install from the README on a Linux x86-64 machine (Ubuntu 22.04, Debian 12 and above):

curl -LO https://github.com/prabanta-dev/janas/releases/latest/download/janas-linux-x86_64.tar.gz
tar xf janas-linux-x86_64.tar.gz && cd janas
./janas-chat

Select a model from the directory, download and verify it, and you're ready to go. Windows 10 (1903+) and 11 have zip builds; the author reports that write speeds on Windows 11 match Linux, and dense model prompt reading is 1.8x faster (sent as whole blocks to the GPU via Intel drivers).

A few practical parameters: --cache <GiB> sets a hard cap on Janas's memory footprint — by default it fills the expert cache with free memory; JANAS_GPU=0 fully disables the GPU (the README candidly notes the GPU path has caused driver resets and outright desktop crashes); janas-bench measures your machine and generates a report, which is currently the most valuable form of feedback for the project. For a first test, close your browser — with a browser and editor open, the same model keeps roughly three-quarters of its speed, but disk wait times increase fivefold.

Limitations You Must Confront

This project is essentially an experiment, not a product. The command-line interface may change several times a week; only the .jns model file format is promised to be stable. All development and testing was done on a single Intel laptop with an integrated Arc GPU — AMD CPUs, discrete GPUs, and slower disks are entirely untested territory. Full memory caching plus full-speed O_DIRECT reads puts heavy strain on a machine, and systems with no headroom may be pushed into swap.

Model availability is the more practical constraint. You need pre-converted .jns files, and currently only the author's Qwen and Gemma catalog on Hugging Face is available — or you convert from GGUF format yourself. Draft acceleration effectiveness depends on text type: on English prose the draft hit rate is 74-89%, but on Italian prose one model managed only about a third. The README also candidly admits that earlier benchmark results were embellished and have since been corrected.

Decision Checklist

If the model fits in memory, stick with llama.cpp. It's mature and general-purpose, and while the small-model gaps are real, they're not significant. If the model exceeds memory and you have a high-speed NVMe SSD, Janas is designed precisely for this scenario — its 24-37 tok/s on an 80B model turns it from a curiosity into a usable tool. If you need stability or specific models, llama.cpp's ecosystem and tooling represent years of accumulation, whereas Janas offers no stability guarantees. If you just want good answers and have internet, paid APIs still beat all local options on a per-token quality basis.

The bigger takeaway: the definition of "too big to run" is shifting from memory capacity to SSD bandwidth. NVMe's 5-7 GB/s streaming read capability already exceeds the speed at which the token generator consumes expert weights. If this trend holds, the effective ceiling for local models will depend on disk rather than memory — and upgrading a disk is far cheaper than upgrading memory.