FreeToken
An open-source inference engine for Mixture-of-Experts (MoE) models, developed by research teams at UC Berkeley and the University of Texas. It optimizes large MoE model inference on consumer-grade hardware through dynamic scheduling strategies and a double-buffering mechanism, positioning itself as an alternative to llama.cpp and Ollama.
Timeline (last 90 days)
FreeToken is an Apache 2.0-licensed edge-native MoE serving engine that keeps non-expert weights on the GPU, stores the expert weight pool in system RAM, and uses remaining VRAM as a dynamic LRU expert cache.
According to the FreeToken technical paper, Qwen3-235B-A3B can achieve 77–83 tokens/s on an RTX 5090 system, and DeepSeek-related models can achieve 22–25 tokens/s on agent trajectories.
FreeToken's current CLI requires Linux x86_64, an NVIDIA GPU, and CUDA 13, with a limited list of supported models.
FreeToken only supports MoE architecture models and requires a high-performance PCIe connection (desktop-grade hardware).
FreeToken delivers maximum value only when the model size exceeds GPU VRAM but is not excessively large; for models that can fit entirely in VRAM, traditional tools remain the better choice.
FreeToken achieves inference speeds 3 times faster than Ollama under the same hardware conditions.
FreeToken supports runtime adjustment of the GPU expert cache size; tests show that reducing the cache from 58% to 40% results in only about 10% performance loss, with performance dropping sharply below 20%.
FreeToken uses a custom FTW weight format, allowing models to be loaded directly from disk into the memory layout required by the engine, skipping the repacking step.
FreeToken's Q* strategy measures the machine's PCIe bandwidth and RAM speed in real time, dynamically deciding whether each batch of missing experts should be transferred from RAM to the GPU or computed directly on the CPU.
FreeToken hides latency through double-buffering: while the GPU computes the current layer, the next layer's expert data is already being transferred via PCIe in the background, achieving parallelization of computation and transfer.
19 more timeline events
All Facts (20)
FreeToken is an Apache 2.0-licensed edge-native MoE serving engine that keeps non-expert weights on the GPU, stores the expert weight pool in system RAM, and uses remaining VRAM as a dynamic LRU expert cache.
50%UnverifiedAccording to the FreeToken technical paper, Qwen3-235B-A3B can achieve 77–83 tokens/s on an RTX 5090 system, and DeepSeek-related models can achieve 22–25 tokens/s on agent trajectories.
50%UnverifiedFreeToken is an open-source inference engine by a research team from Berkeley and the University of Texas, deeply optimized for MoE models on consumer-grade hardware, serving as an alternative to LLAMA.CPP or Ollama.
50%UnverifiedFreeToken achieves inference speeds 3 times faster than Ollama under the same hardware conditions.
50%UnverifiedFreeToken treats GPU VRAM as a cache, with the complete model always stored in system memory as the source of truth, and the GPU only retains the most frequently used experts.
50%UnverifiedFreeToken hides latency through double-buffering: while the GPU computes the current layer, the next layer's expert data is already being transferred via PCIe in the background, achieving parallelization of computation and transfer.
50%UnverifiedFreeToken's Q* strategy measures the machine's PCIe bandwidth and RAM speed in real time, dynamically deciding whether each batch of missing experts should be transferred from RAM to the GPU or computed directly on the CPU.
50%UnverifiedFreeToken uses a custom FTW weight format, allowing models to be loaded directly from disk into the memory layout required by the engine, skipping the repacking step.
50%UnverifiedFreeToken supports runtime adjustment of the GPU expert cache size; tests show that reducing the cache from 58% to 40% results in only about 10% performance loss, with performance dropping sharply below 20%.
50%UnverifiedIn tests with the 4-bit quantized version (fitting entirely into 32 GB of VRAM), Ollama reached 240 tokens/s while FreeToken reached 225 tokens/s, making Ollama actually faster.
50%UnverifiedFreeToken delivers maximum value only when the model size exceeds GPU VRAM but is not excessively large; for models that can fit entirely in VRAM, traditional tools remain the better choice.
50%UnverifiedFreeToken only supports MoE architecture models and requires a high-performance PCIe connection (desktop-grade hardware).
50%UnverifiedThe FreeToken desktop application currently only supports Windows and Linux, offers model compatibility detection, but cannot access project folders or modify files, making it more suitable for use via CLI.
50%UnverifiedFreeToken实测8GB显存笔记本可运行约35B参数模型
50%UnverifiedFreeToken使753B参数的超大模型可以在单张工作站显卡上运行
50%UnverifiedFreeToken论文并未披露生成速度（tok/s）和延迟数据，仅证明模型能够加载和推理
50%UnverifiedFreeToken能让753B参数的模型在单张工作站显卡上运行
50%UnverifiedFreeToken在8GB显存笔记本上可运行约35B参数模型，游戏级桌面显卡可运行284B参数模型，工作站单卡可运行753B参数模型
50%UnverifiedFreeToken目前支持20多个开源混合专家模型
50%Unverified参数在显存、内存、磁盘之间来回搬运会带来显著的I/O开销，可能导致实际生成速度降至0.5 tok/s以下
50%