WebLLM

WebLLM is a high-performance in-browser LLM inference engine that aims to move the entire model inference pipeline to the client-side browser for execution, with no server support required.

Go Outside runs models via WebLLM (@mlc-ai/[email protected]), an inference engine open-sourced by MLC that can compile models to WebGPU.

The WebLLM project has achieved in-browser execution of mainstream small models including Llama 3, Phi-3, and Gemma.

On devices equipped with a discrete GPU, the inference speed of small models running in the browser can reach tens of tokens per second.

Projects such as Transformers.js and WebLLM have demonstrated that small-to-medium-scale language models can run smoothly in the browser without a backend server.

In the WebLLM architecture, Wasm primarily handles non-GPU computation logic, including tokenizer encoding and decoding, sampling strategy execution, KV Cache management, and other CPU-side tasks, forming a GPU+CPU collaborative inference pipeline with WebGPU.

GQA reduces the size of the KV Cache by allowing multiple Query heads to share the same set of Key-Value heads, which is particularly critical for WebLLM operating in memory-constrained environments.

The WebLLM project typically uses the MLC-LLM framework for model quantization and compilation optimization, with Apache TVM serving as the underlying infrastructure for MLC-LLM.

Once the model is fully loaded, WebLLM applications can continue to function in an offline state, and combined with PWA technology, they can be installed to a device's home screen like a native application.

Even after quantization, a usable LLM weight file typically still ranges from hundreds of MB to several GB, which is a limitation of WebLLM's initial load time.

In the WebLLM architecture, Wasm handles the non-GPU computation logic during model inference, including CPU-side tasks such as tokenizer encoding and decoding, sampling strategy execution, and KV Cache management.

In-browser inference shifts model computation costs to the user side, freeing developers from paying for GPU inference servers and enabling the construction of fully backend-free AI applications.

WebLLM leverages built-in browser storage mechanisms (such as the Cache API and IndexedDB) to persist downloaded model weights, avoiding repeated downloads.

The VRAM that WebGPU can allocate is constrained by the browser sandbox, typically far less than what native applications can access, which is an important reason for the performance bottleneck WebLLM faces with long-context tasks.

WebLLM is better suited for privacy-sensitive assistant tools, educational and demonstration scenarios, and lightweight embedded AI features, rather than scenarios requiring very large parameter models, extreme response speeds, or broad coverage on low-end devices.

WebLLM keeps all inference local, with user data never leaving the device throughout the entire process, fundamentally eliminating the risk of data leakage.