Running RAG in the Browser: Field Tests and Reflections on Small Language Models

Running a language model in the browser is no longer novel, but moving the entire retrieval-augmented generation (RAG) pipeline client-side is an interesting engineering experiment. A developer did exactly this on their blog: the search finds passages in your browser, a small model under 600MB reads those passages on your GPU and generates a two-sentence summary — all without any network connection or data upload.

This article doesn't aim to repeat the "so cool" praise, but rather to break down how it works technically, and why small models doing RAG have surprisingly low faithfulness.

How RAG Runs in the Browser

The entire pipeline runs client-side, in four steps. Retrieval: The user's question first goes through a search based on static embeddings, hitting the four most relevant passages. This search itself uses only a lookup table of about 4MB, no server needed. Gating: If the highest-scoring passage falls below a threshold, the model isn't invoked and a message like "this content isn't found on the blog" is shown. Prompt construction: The four numbered passages are added to the system prompt, instructing the model to answer in at most two sentences, using only these passages, and to say so directly if it can't answer. Generation: A Web Worker runs the model on WebGPU; browsers without WebGPU support fall back to WASM on CPU, with results streamed and stopping once two sentences are complete.

The entire process requires no API keys and no accounts. The model is downloaded once (a few hundred MB) and cached locally, no repeated downloads afterward. The author also provides a button to clear the cache, leaving the choice to the user.

The difference between WebGPU and WASM deserves elaboration: WebGPU is the modern graphics and compute API exposed by browsers, letting JavaScript directly call the GPU for parallel matrix operations, well-suited for accelerating large model inference; WASM (WebAssembly) is a binary instruction format that runs in the browser, primarily on the CPU, with broader compatibility but much slower speed. The fallback strategy in the article is based on this reality: use the GPU if WebGPU is supported, otherwise fall back to CPU running WASM, at the cost of significantly reduced generation speed.

A Web Worker is a background thread mechanism provided by browsers to offload heavy tasks like model inference from the main thread, preventing the page from freezing. The model loads weights and runs inference inside the Worker, streaming tokens back to the main thread for rendering via message channels. This is also why the pipeline can "display as it generates" rather than waiting for the entire answer to be computed.

Model Selection: Small Size Doesn't Mean Usable

The author listed a batch of candidate models with ONNX browser builds: SmolLM2 (135M and 360M parameters), Gemma 3 270M, LFM2.5-350M, Qwen2.5-0.5B and Qwen3-0.6B. As references, two "would never ship" options were also included: Gemma 4 E2B requiring a 3.1GB download, and Claude Haiku 5.5 via API.

During selection, a counterintuitive discovery emerged: for several models, the version labeled "4-bit" (q4) was actually larger than the q4f16 version, because the q4 export keeps the embedding table at full 32-bit precision. Taking SmolLM2-360M as an example, q4 is 390MB while q4f16 is only 275MB, yet their outputs are identical. So all tests used q4f16.

Evaluation Design

The evaluation reused 30 questions from an earlier semantic search experiment, rewritten as genuine interrogative sentences, plus 15 additional questions the blog genuinely couldn't answer. Most of these "trick questions" weren't completely off-topic but rather borderline: e.g., "which GPU should I buy to run Llama 70B locally," "how is BM25 better than TF-IDF," "how to fine-tune Kokoro to speak in my own voice." The blog discussed related topics but didn't provide answers to these questions.

Each model received the same four passages and the same prompt. Opus 5.5 served as judge, labeling each answer as faithful (all claims can be found in the passages), unsupported (at least one sentence isn't in the passages, even if coincidentally true), or no answer. The author independently annotated 20 entries without knowing the judge's conclusions, 16 of which agreed with the judge on the "faithful or not" judgment.

ONNX (Open Neural Network Exchange) is a cross-framework model exchange format. Models trained in PyTorch, TensorFlow, etc., can be exported as ONNX graphs and then loaded by different runtimes. The browser-side transformers.js executes these models through ONNX Runtime Web, which is why candidate models must have "ONNX browser builds."

q4 and q4f16 are two quantization configurations. q4 typically refers to compressing weights to 4-bit integers, but some export implementations keep the embedding table as FP32; q4f16 additionally converts the embedding table to FP16, making the file smaller. The embedding table is the mapping from vocabulary to vectors, with considerable parameter count, and its precision choice directly determines model size. This also explains the counterintuitive phenomenon of "the version labeled 4-bit being larger" mentioned in the article.

Small Model Performance: Confidently Wrong

The results are straightforward: ready-made models under 350MB are essentially unusable. They're not just slightly worse — across 30 answerable questions they achieved faithfulness at most 3 times, while eagerly answering almost any question.

Ask "which GPU should I buy to run Llama 70B locally" while feeding it four passages about embeddings and MacBook coding agents, and you get answers like:

"...the best option to run Llama 70B locally is likely to be a NVIDIA GTX 960 or GTX 970." (SmolLM2-360M)

"...a high-performance graphics card (GPU) such as a NVIDIA A100 series (e.g., RTX 3090, RTX 3080, 3070, 3080, 3070M, 3080M)" (Gemma 3 270M)

"...popular options include RTX 3090, RTR 5900, or newer generations like TensorRT 60+" (LFM2.5-350M)

Gemma 4 E2B correctly pointed out that "the provided passages do not contain information about which GPU to buy for running Llama 70B locally."

Even on questions the blog could genuinely answer, small models often mangle the content. LFM2.5-350M explained Bloom filters quickly and fluently, but described the bit array as a hash table, essentially fabricating the content. That said, let's be fair to the model: the search didn't hand it the passage explaining Bloom filter principles — it got passages about use cases and interactive demos. The quality ceiling of the summary is the quality of the passages retrieval gives it.

Getting Stuck and Repetition Penalty

On first runs, most small models' answers weren't so much wrong as stuck. Asked how to add OpenSearch support to a website, SmolLM2-135M's output became an infinite repetition of "Search from your URL bar"; Gemma 3 270M simply spat out a long string of numbers until exhausting its tokens.

This is a loop small models easily fall into when "always picking the most likely next token." A simple fix is adding a repetition penalty (1.15 in this case), but the author explicitly points out: it only makes the model write longer, more varied nonsense, without improving faithfulness or quality.

Repetition penalty is a sampling correction during decoding: it lowers the probability of tokens that have already appeared, thereby suppressing loops. It typically works alongside temperature, top-p, and other parameters; setting it too high makes output rigid or even semantically broken. The 1.15 in the article is a mild setting, which is why the author says it merely makes the model "write longer, more varied nonsense" — it changes surface diversity, not whether the model answers based on the passages.

Token exhaustion refers to the model being forced to stop after reaching the set maximum generation length. When small models fall into loops of numbers or phrases, they often only stop after exhausting the entire token budget, presenting as a long output with zero information content.

Limitations of Relevance Gating

Using relevance scores for gating to block fabrication looks like a cheap solution. It does work for far-off-topic questions: "how to bake sourdough bread at home" scored 0.25, below the 0.45 threshold, and was directly blocked — which is necessary, since Gemma 3's answer to that question was a bread recipe.

But borderline questions score 0.54 to 0.63, higher than several questions the blog could actually answer, because they hit articles on the same topic. To block these too, the threshold would have to falsely kill 13 of the 30 real questions.

Gating can only block clearly off-topic inputs; everything else rests on the model. This is exactly why model size matters here: Gemma 4 and Haiku will say "the passages don't cover this question," while small models never refuse to answer.

Renting GPUs vs. Using APIs

From a cost perspective, using APIs is more economical in the vast majority of cases. At the time's pricing, the cheapest suitable cloud GPU (Google Cloud's NVIDIA L4) was about $0.71/hour, roughly $516 for non-stop operation all month. Based on the blog's monthly traffic estimate, even if every reader asked one question, the cost per Q&A would exceed one dollar. Meanwhile, Haiku answering all 45 evaluation questions cost only $0.006.

The author's advice is straightforward: if you have a real use case, start with Haiku. Self-hosting is only justified under hard constraints like compliance, privacy, or offline deployment. Another practical reminder: don't put API keys in static sites, to avoid having your bill raided.

A static site refers to a website containing only static resources like HTML, CSS, JavaScript, distributed directly by CDN, with no server-side location to safely store keys. Writing an API key into the front-end code of such a site is equivalent to exposing it to anyone who opens developer tools, risking misuse and high bills. This is another layer of motivation behind the browser-side approach's emphasis on "no network, no data upload": it naturally sidesteps the key custody problem.

Longer Answers Are Less Safe

If Qwen3's two sentences are only half right, would letting it say more be better? The author compared Qwen3-0.6B and Gemma 4 E2B, allowing answers of 2, 4, or 6 sentences.

Both became less faithful as space widened, but the small model dropped more steeply: Qwen3's faithful answers went from 15 to 9 to 5 (out of 30). Every extra sentence is another chance to fabricate. By 6 sentences, it simply stopped refusing the GPU question, instead summarizing every passage and ending with "the answer depends on whether the user needs a high-performance GPU or an ordinary GPU." Gemma 4 also declined slightly (28, 26, 24), but it rarely used the extra space (about 2.8 sentences when allowed 6) and consistently refused to fabricate answers to questions the blog didn't cover.

Next Steps and Conclusion

The author's next step is to make a 300MB-class model do this single task better: it doesn't need to know everything, just read four passages and summarize or honestly say "can't answer." This is more of a fine-tuning problem. Another idea is to quietly use the already-loaded model for homepage search, further reducing additional downloads.

The conclusion from this experiment is plain but important: the engineering pipeline for browser-side RAG already works, but retrieval quality determines the ceiling, and small model faithfulness determines trustworthiness. The correct way to use such pages is to first look at the retrieved passages, then decide whether to trust those two sentences.

One-line Summary

Browser-side RAG pipeline works, but small model faithfulness is extremely low; retrieval quality determines the ceiling.

Paragraph Summary

The article documents an engineering experiment moving the full RAG pipeline to the browser: retrieval, gating, prompt construction, and generation all happen client-side, with the model running via transformers.js on WebGPU or WASM, and data never leaving the machine. The author evaluated small models including SmolLM2, Gemma 3 270M, and Qwen3-0.6B using 30 answerable questions plus 15 trick questions, with Gemma 4 E2B and Haiku as controls. The conclusion: ready-made models under 350MB are nearly unusable, achieving at most 3 faithful answers out of 30, and almost never refusing to answer; relevance gating only blocks clearly off-topic questions, while borderline questions actually score higher; allowing more output sentences further lowers faithfulness. Cost-wise, APIs far outperform self-hosted cloud GPUs.