Running LLMs in the Browser: WebGPU Hits 63 tokens/s, WASM Only 1.6

Running language models inside a webpage is no longer just a demo. A chatroom project called BEINGS ran a full real-world test: the same 360M-parameter small model hit 62.9 tokens/s on the WebGPU backend, while falling back to the CPU-based WASM backend gave only 1.6 tokens/s—with median reply times of 0.54 seconds and 26.9 seconds, respectively.

What does that gap mean? In a chat setting, 27 seconds means the conversation has long moved on while your AI character is still finishing its sentence. But for a platform that doesn't want to pay a cent for inference, putting the compute on users' own browsers is still a good deal.

Why Put Inference on User Devices

BEINGS is built around the idea that everyone brings their own AI character into the chatroom: a finance bro who thinks pizza is a market signal, a medieval knight who treats phones as sorcery. These characters talk to each other and to people around the clock, and the call volume is substantial.

If you routed that through a server to a paid API, every busy room would be a bill. So the architecture flips: each character runs on the device of the person who brought it in. The landing page says it plainly—no API key, no signup, no payment. Cloud brains (OpenAI, Claude, Gemini) are an optional upgrade: users paste their own key, which lives in localStorage and connects directly to the provider, never passing through the platform.

The trade-offs are honest too: users have to download model files ranging from a few hundred MB to over 1GB, their local GPU does the work, and small models are indeed much dumber than large cloud models. For a chaotic chatroom, dumb-but-free is the right trade; for a customer service bot, probably not.

Core Implementation: Worker + Transformers.js + WebGPU

Running an LLM in the browser needs three things: a model converted to ONNX and quantized (4-bit weights, about 1.2GB for a 1B model instead of 5GB), a runtime that executes it in the page (Transformers.js, which wraps ONNX Runtime Web), and a GPU backend (WebGPU), with WebAssembly on the CPU as the fallback. The model must live in a Web Worker, or the page will freeze.

The code is about 90 lines of JavaScript. A few details worth noting:

• dtype follows the device: load q4f16 on WebGPU (4-bit weights, 16-bit compute) and q4 on WASM. These are two different files on Hugging Face—SmolLM2's WebGPU file is 272.7MB and its WASM file is 387.9MB.
• WebGPU being present doesn't mean it works: "gpu" in navigator being true doesn't mean the driver cooperates, so when WebGPU loading fails, automatically retry with WASM instead of erroring out.
• Progress must be reported back to the page: a 1.2GB first download with no progress bar looks like a broken website.
• Transformers.js is loaded from a pinned-version CDN, not from npm—deliberately.

A Silent Bundler Trap

BEINGS is a Next.js 16 + Turbopack project. Using the "standard" approach (a TypeScript worker file + new Worker(new URL(...))), Turbopack emitted the worker as a raw .ts file into the static directory. Browsers can't execute TypeScript, the worker never started, and all local inference was dead—with zero build errors.

The fix is unglamorous: write the worker by hand as JavaScript, put it in /public, serve it as-is, load it by a plain path, pull in Transformers.js from the CDN, and keep the bundler entirely out of it. The lesson: after every build, go to the deployed site and confirm the worker actually starts—a silently dead worker looks exactly like a slow model.

One Generation at a Time

A single user might bring six characters into the same room. If all six called the model at once, a laptop would either load six copies or fight over one GPU. So the page holds exactly one worker, and all generation requests run through a serial queue.

The key is how the queue chains: each call waits for the previous one to finish, whether it succeeded or failed, so one crashed generation doesn't block the whole line. The 90-second timeout exists because a single reply on the WASM fallback can take nearly a minute.

ONNX and quantization: ONNX (Open Neural Network Exchange) is a cross-framework model exchange format; models trained in frameworks like PyTorch can be exported as ONNX computation graphs, which ONNX Runtime then executes on a variety of hardware. Transformers.js in the browser runs on ONNX Runtime Web underneath. Quantization compresses model weights from 16-bit floats to 4-bit integers, shrinking size to about a quarter at the cost of some accuracy. The q4f16 mentioned above means 4-bit weights dequantized to 16-bit floats for compute, which suits WebGPU's floating-point units; q4 executes entirely on an integer path, which suits WASM's CPU implementation. The same model therefore has multiple weight files of different sizes, and which one to pick depends on the target backend.

How WebGPU and WebAssembly divide the work: WebGPU is the new standard exposing low-level graphics and compute APIs in the browser, executing massively parallel matrix operations via WGSL shaders—currently the performance path for running LLMs on the web. WebAssembly (WASM), meanwhile, is portable bytecode that executes on the CPU at near-native speed but lacks the GPU's parallel throughput. In Transformers.js the two are switchable backends: prefer WebGPU, fall back to WASM on failure—which explains the two-orders-of-magnitude performance gap described here.

Measured Data: The Gulf Between WebGPU and WASM

The test environment was Chrome 155, Windows 11, an RTX 4070 desktop, and 24 logical cores. The script opened its own tab on the live site, started the real worker, loaded each model, and asked 24 real chat turns with a 70-token cap per turn.

The headline is about the backend, not the model:

Model	WebGPU	WASM
SmolLM2 360M	62.9 tokens/s, 0.54 s/reply	1.6 tokens/s, 26.9 s/reply
Llama 3.2 1B	55–64 tokens/s, 0.4–0.5 s/reply	not tested
Qwen 2.5 0.5B	37–40 tokens/s, ~1.8 s/reply	not tested

The 1.6 tokens/s on WASM may also be a self-inflicted wound. ONNX Runtime Web only uses multithreading when the page is cross-origin isolated; otherwise it runs single-threaded. BEINGS's page sends COOP headers but not COEP headers, and the benchmark tab reported crossOriginIsolated: false with no SharedArrayBuffer—meaning the fallback path used only 1 thread on a 24-thread machine. The author suspects enabling isolation could make the fallback several times faster, but hasn't measured it, so makes no promises. The trouble is that COEP blocks cross-origin embeds that don't declare permission, and BEINGS embeds YouTube and Wikimedia content.

There's also a counterintuitive finding: the biggest model replied fastest. Llama 3.2 1B generated at a speed close to SmolLM2 but took only 0.4–0.5 seconds for a median reply, because it finished in 24 to 30 tokens; the smaller models rambled all the way to the 70-token cap. Qwen 0.5B was the slowest of the three.

Download Is the Real Cost of the First Play

Cold starts (including download) were 7.5 seconds (SmolLM2, 273MB), 13.4 seconds (Qwen, 483MB), and 26.8 seconds (Llama, 1238MB). Once cached, the same loads take just 1.0, 2.2, and 3.1 seconds. Transformers.js stores files in the browser's Cache Storage, so the second visit is fast without writing any caching code.

But not every user has fast internet. 1.2GB over mobile 4G means "come back later," which is why the app lets users pick a smaller brain—and why progress reporting matters more than it looks.

Cross-origin isolation and SharedArrayBuffer: WASM multithreading depends on SharedArrayBuffer, which browsers only allow when the page is "cross-origin isolated," out of concern for side-channel attacks like Spectre. Enabling isolation requires sending two response headers: COOP (Cross-Origin-Opener-Policy: same-origin) and COEP (Cross-Origin-Embedder-Policy: require-corp). COEP requires every cross-origin resource embedded in the page to explicitly declare permission (via CORP headers or CORS), or it gets blocked. BEINGS sends COOP but not COEP, so crossOriginIsolated is false and ONNX Runtime Web falls back to single-threaded—part of the reason for the 1.6 tokens/s. The cost of enabling isolation is handling third-party content like YouTube embeds and Wikimedia images one by one, which is why the author didn't just turn it on.

Sampling Parameters Affect Perceived Quality More Than Model Size

Early versions used temperature 1.05 and top_p 0.95 in pursuit of "more chaos." The result: even Qwen looped on "u.u.u.u.u." and "The The FQ Q M". The configuration that finally stabilized was: temperature 0.75, top_p 0.9, top_k 40, repetition penalty 1.12, plus no_repeat_ngram_size: 3 (forbidding any three-token sequence from repeating).

Only temperature was tested this time; the rest stayed at safe values:

• Qwen 0.5B: at 0.75, 2 of 12 were discarded as garbage; at 1.05, 6 of 12 were discarded. A real sample from high temperature: "CODESUPONSEGRAPHTEGOINSTRUMENTPRODUSONGRAPH..."
• SmolLM2 360M: median reply length grew from 15 words to 40.5 words, and 7 of 12 hit the 70-token cap. The prompt asked for under 20 words; at high temperature it stopped listening.
• Llama 3.2 1B: all 24 usable at both temperatures, and stayed in character.

If you change only one setting: keep temperature at or below 0.8 for models under 1B parameters, and turn on no_repeat_ngram_size.

What each sampling parameter does: temperature controls how smooth the probability distribution is—higher values make output more random, values near 0 approach greedy decoding; top_p (nucleus sampling) samples from the smallest set of words whose cumulative probability reaches p, truncating the low-probability long tail; top_k keeps only the k highest-probability candidate words. Repetition penalty lowers the probability of tokens that have already appeared, while no_repeat_ngram_size is a harder constraint: it outright forbids any token sequence of length n from repeating, with n=3 banning three-word repeats. Because of their limited capacity, small models easily fall into local loops in the probability distribution at high temperatures, showing up as degenerate output like "u.u.u.u."; lowering the temperature and adding a hard n-gram constraint improves perceived quality more directly than switching to a bigger model.

Garbage Filter: Silence Beats Nonsense

Even with tuned parameters, the 360M model occasionally produces things nobody should see. BEINGS's rule is "better a silent character than a broken one," so every reply passes through a deterministic rule-based filter before posting—no second model, no API call, 64 lines of code.

The filter checks for: markup or template fragments, symbol soup with under 55% letters, short-fragment repetition ("u.u.u.u."), lexical loops, stray single- and double-letter fragments, and coined words with no vowels. It has its own test file, using real bad lines from bug reports as fixtures, plus good lines that must pass, such as "Penar wingle berry" and "THIS IS BULLISH ACTUALLY, LOAD UP". The result: 14 passed, 0 failed.

What It Caught and What It Missed

Running all 96 benchmark replies through the filter and an instruction-leak check discarded 13: 11 judged garbage, 2 judged leaked instructions (one SmolLM2 reply ended with "Speak in character now!"). Stuck-together words, stray --> markers, and Qwen's word salad were all caught.

But it also let through things a human would throw out. Qwen's "Toestrafing GRating" passed because both words have vowels and neither repeats; an all-caps SmolLM2 reply about "MULTILATERAL ORTHODOZIC" passed too. A rule-based filter catches broken text, not meaningless text. The author's judgment: fine for a chaotic chatroom, nowhere near enough for anything a human will actually read—if semantics matter, you need a bigger model, not a smarter regex.

Pitfalls and Practical Advice

The author reviewed several failures: the bundler emitting the worker as .ts silently killing inference; chaotic sampling parameters sending small models in circles; small models parroting their own instructions (fixed by leak filtering and switching prompts to real chat-turn format, because "small models parrot meta-instructions, but can handle the chat format they were fine-tuned on"); six characters fighting over one laptop; and never having validated on real hardware—until this test.

For anyone wanting to do the same in their own project: use a module Web Worker to host Transformers.js, one worker, one model, serial generation; pick ONNX models that ship both q4f16 and q4 files; default to around 1B parameters, not the smallest model you can load—in the tests, Llama 3.2 1B replied as fast as SmolLM2 360M and was far more coherent, so leave small models for the "slow internet" option; do the WebGPU-to-WASM fallback inside the worker, and if fallback performance matters, enable cross-origin isolation, then test whether your embeds still work; keep temperature at or below 0.75 for small models; filter before publishing; show download progress and disclose size upfront; and after every build, confirm in a real browser that the worker is alive.

Where This Could Go

The author argues the interesting shift isn't that small models got good—they haven't. It's that "the user's device pays" changes what's worth building. A feature that costs a cent per message on an API is free forever in the browser, which makes a whole class of silly, chatty, always-on features viable for indie developers. Hardware keeps moving the line too: the 1B model in today's browser tab is roughly the scale that needed a server not long ago. At the other end, an "open" 816B model would need about 1TB of memory to run.

As for how to choose: if your app really does call a hosted model, the cheapest token is the one you don't send twice; and if you're deciding between in-browser inference and an API, work out the API side's costs first.

One-line Summary

In-browser LLM benchmark: WebGPU reaches 62.9 tokens/s, WASM fallback only 1.6, and the 1B model is actually the best value.

Paragraph Summary

The BEINGS chatroom project runs each AI character in the browser of the user who brought it into the room, eliminating server inference bills. In tests, the same 360M model hit 62.9 tokens/s with 0.54 s per reply on WebGPU, versus just 1.6 tokens/s and 26.9 s on the WASM fallback—partly because the page isn't cross-origin isolated and runs single-threaded. The implementation relies on quantized ONNX models, Transformers.js, and WebGPU; the model must live in a Web Worker, and you must watch out for Turbopack emitting a TS worker as a static asset, which fails silently. Counterintuitively, the 1B Llama 3.2 replies as fast as the small models and is more coherent, since small models ramble to the token cap. Sampling parameters affect perceived quality more than model size, and a rule-based garbage filter catches broken text but not meaningless text.