FreeToken Inference Engine: 3x Speed Boost for MoE Models

FreeToken: An Inference Engine Optimized for MoE Models

Running large Mixture-of-Experts (MoE) models locally has long been bottlenecked by insufficient VRAM. A research team from UC Berkeley and the University of Texas has open-sourced FreeToken, an inference engine that achieves up to 3x faster inference than Ollama on the same hardware through an innovative scheduling strategy.

FreeToken is essentially an alternative to LLAMA.CPP or Ollama, but deeply optimized for MoE models on consumer-grade hardware. Take DeepSeek V3 Flash as an example: the model has 285B total parameters, but only activates 13B per token. While the active parameters fit in the GPU, the full model still requires 285B of storage — creating a massive memory bottleneck.

From Static Allocation to Dynamic Scheduling

Traditional tools like LLAMA.CPP use a fixed layer-assignment strategy: pinning certain layers to the GPU at load time while placing the rest on the CPU. But MoE models change which experts they use with every token, so a fixed layout misses most hot data and degrades performance.

FreeToken reframes this as a scheduling problem rather than a placement problem. It treats GPU VRAM as a cache, keeps the full model in system RAM as the source of truth, and only retains the most frequently used experts on the GPU. Research shows that expert usage exhibits strong locality — consecutive tokens tend to call the same experts repeatedly — so most requests end up as cache hits.

Dual-Buffering to Solve the Prefill Bottleneck

When processing long prompts, sparsity advantages disappear because nearly all experts must be traversed. FreeToken hides this latency with a dual-buffering technique: while the GPU computes the current layer, the next layer's expert data is already being transferred over PCIe in the background — parallelizing compute and data transfer.

Q* Strategy: Intelligently Choosing Where to Compute

On a cache miss, the system faces two options: transfer the expert from RAM to the GPU, or compute it directly on the CPU. FreeToken's Q* strategy continuously measures the machine's PCIe bandwidth and RAM speed, dynamically deciding how to handle each batch of missing experts. A laptop's narrower PCIe bus and a desktop's faster RAM automatically receive different optimized strategies — no manual configuration needed.

FreeToken also uses a custom FTW weight format that lets models load directly from disk into the memory layout the engine requires, skipping any repacking step and dramatically reducing startup time.

RTX 5090 Benchmark: Performance When VRAM Is Exceeded

Test environment: RTX 5090 (32GB VRAM) + 64GB DDR5 RAM. The QWEN 3.6 35B model at 8-bit quantization (~38GB) was loaded, exceeding VRAM by 6GB.

How Ollama handles it:

• 70% of the model stays on the GPU, 30% is offloaded to CPU
• Every token must pass through all layers, forcing traversal through slower CPU paths
• Inference speed: 58 tokens/s
• Time to complete the test task: 14 minutes 20 seconds

How FreeToken handles it:

• All computation stays on the GPU; only missing experts are streamed over PCIe
• Inference speed: 132 tokens/s
• Time to complete the same task: 4 minutes 40 seconds
• Performance gain: 2.3x

Dynamic Cache Adjustment Experiments

FreeToken supports adjusting the GPU expert cache size at runtime without restarting the server. Tests revealed:

• Dropping from 58% cache to 40% costs only ~10% performance
• Below 20% cache, performance drops sharply as PCIe bandwidth becomes the bottleneck
• This validates the theoretical assumption that "a small number of hot experts do most of the work"

Use Cases and Limitations

Where FreeToken excels:

• Model size exceeds available GPU VRAM
• Only supports MoE architecture models
• Requires high-performance PCIe connectivity (desktop-class hardware)

A case where performance reverses: When testing a 4-bit quantized version (which fits entirely within 32GB VRAM):

• Ollama: 240 tokens/s
• FreeToken: 225 tokens/s

Here, Ollama loads the entire model into the GPU and wins. FreeToken's streaming architecture introduces engineering overhead when no streaming is needed.

Current state of the desktop app: Only Windows and Linux are currently supported, with a model compatibility check feature. However, testing found it couldn't access project folders or modify files — functionality remains limited, making CLI usage the more practical approach.

Technical Significance and Future Outlook

FreeToken's core value lies in its adaptability: it automatically analyzes hardware configuration and generates an optimal streaming strategy for each machine's PCIe bandwidth and RAM speed combination. This dynamic scheduling approach opens new possibilities for running very large MoE models on consumer hardware.

That said, this is not a silver bullet. FreeToken delivers maximum value only when a model falls into the sweet spot of "exceeds VRAM but doesn't massively overwhelm the hardware." For models that fit entirely in VRAM, traditional tools remain the better choice.

As large MoE models like DeepSeek V3 become more widespread, targeted optimization tools like FreeToken will grow increasingly important. The open-source release gives the community a valuable reference implementation, and the underlying scheduling strategy is well worth deeper study.