Zero-Token Decisions: EmbeddingGemma 2 + WebGPU for In-Browser Multimodal Classification

When a Decision Shouldn't Be a "Generation"

Most people who use large language models for classification tasks instinctively ask them to "say something": Is this egg fertilized? Can this scene continue? Which of four states are we currently in? But using a generative chat model to answer these kinds of questions is fundamentally the wrong tool for the job. Generation is slow, non-deterministic, and every token produced is an opportunity for the answer to drift.

A decision has a type — a category, a yes/no, a score. So why not get that type directly?

A static web page called Decide Live is built around exactly this idea. It takes a camera feed, a microphone window, an image, or a line of text and converts it directly into a typed decision, entirely within the browser. It runs a fine-tuned EmbeddingGemma 2 (Apache-2.0 license) via transformers.js on WebGPU, generates zero tokens, and uploads no data.

The author calls this local paradigm ZTC (Zero-Token Confidence). No server, no per-call API fees — model weights are downloaded once and cached.

How a Decision Is Computed

Decide Live's decision pipeline is fully documented on the page and in the model card, making it reproducible. The entire flow involves no decoding loop whatsoever; it comes down to four steps:

Query Embedding and Option Embeddings

First, the query embedding. A media placeholder comes first, followed by task: classification | query: {question}, with \nContext: {context} appended when context is provided. The model embeds both the media and text in a single forward pass.

Option embeddings format each option as title: none | text: {description}, embed it once, and cache the result.

Embedding models differ fundamentally from generative models: instead of producing a new text sequence, they map inputs (text, image, audio) to a point in a high-dimensional vector space. Semantically similar inputs produce vectors that are closer together in that space. EmbeddingGemma 2 is a variant of Gemma 2 fine-tuned with contrastive learning, with the goal of pulling the "query" vector close to the correct option and pushing it away from incorrect ones. This follows the same paradigm as CLIP and sentence-transformers. The key advantage: embedding requires only a single encoder forward pass — there is no autoregressive decoding loop. The computation is deterministic and bounded, which is precisely why it can run in real time inside a browser.

Pooling and Scoring

Mean pooling is applied over the attention mask, followed by L2 normalization, yielding a single 768-dimensional unit vector. The scoring formula is:

P(option_i) = softmax_i( 19.96 * cos(e_query, e_option_i) )

The scaling factor 19.96 is learned (corresponding to a temperature of ~0.05) and is published alongside decision.json. The entire pipeline is one encoder forward pass per card, plus a dot product — no per-token generation.

This is exactly what "zero token" means: the decision is derived from a softmax over cosine similarities between the query embedding and option embeddings, with no text generation involved. The result is highly deterministic and extremely cheap to compute.

Mean pooling is the standard approach of averaging the hidden states of all tokens in a sequence, weighted by the attention mask, into a single vector. Compared to using only the [CLS] token, mean pooling typically captures global semantic information more uniformly across the sequence. L2 normalization then scales the vector onto the unit sphere, making cosine similarity equivalent to a dot product for more efficient computation.

The scaling factor 19.96 in the formula (equivalent to a temperature parameter τ = 1/19.96 ≈ 0.05) controls the "sharpness" of the softmax: a lower temperature concentrates probability mass on the highest-scoring option, yielding more decisive outputs; a higher temperature flattens the distribution, expressing more uncertainty. This coefficient is learned during training rather than hand-tuned, reflecting the model's self-calibrated confidence.

The In-Browser Stack

Decide Live's engineering implementation is equally noteworthy, demonstrating a complete and viable approach to on-device AI today:

• transformers.js 4.3.1 loads the model inside a Web Worker and returns embeddings, avoiding main-thread blocking.
• WebGPU executes the ONNX compute graph; the system automatically falls back to WASM (CPU) when WebGPU is unavailable.
• ONNX files are provided as a q4f16 build and a portable q4 build (4-bit weights with standard 8-bit gather, so the same files also run on WASM runtimes that lack GatherBlockQuantized support), plus a 4-bit audio encoder. The fp16 and fp32 variants are omitted to fit within Hugging Face Space's 1 GB limit.
• Weights are downloaded once and cached by the browser. All subsequent decisions run entirely locally.

This design means the first visit requires a model download, but afterward the tool runs fully offline, independently of any server.

WebGPU is a next-generation browser graphics and compute API officially standardized by the W3C in 2023. Compared to WebGL, it offers much closer-to-native GPU compute capabilities (via Compute Shaders), enabling tensor operations like matrix multiplication to run efficiently directly in the browser. transformers.js is Hugging Face's JavaScript inference library, reusing model weights (in ONNX format) from the Hugging Face Hub and wrapping preprocessing steps like tokenization and feature extraction. Together, they allow Transformer models that previously required a Python/PyTorch environment to run fully inside an ordinary browser tab with no server needed. q4f16 means weights are stored in 4-bit quantized form and dequantized to fp16 at inference time — the current mainstream precision trade-off for on-device inference, offering a good balance between model size and accuracy.

What It Can Actually Do

Decide Live offers several modes you can try directly:

• Live mode: Embeds the camera feed and scores up to four decision cards, each with its own question, using the "one forward pass per card" format from training. Each card displays a probability bar, the current decision, and a confidence gate.
• Microphone / audio file: The same card mechanism applied to 16 kHz mono audio in 2-second windows; quiet windows below -55 dBFS are skipped.
• Single input: One image, one text passage, one audio clip, or a short video, paired with one question and 2–8 options written as key: description pairs, or a yes/no question.
• Sample tests: 18 image tests with ground-truth answers, runnable in the browser, producing an accuracy report and a comparison of different prompt formats.

Why This Matters for Edge AI

A decision running on-device inherently carries three advantages: privacy (camera feed and audio never leave the device — privacy is guaranteed by construction, not by promise), zero marginal cost (every call is free), and offline availability (works without a network connection after the first load).

Combine these properties with the zero-token approach, and you can score every frame or every audio window without incurring cloud bills or network round-trip latency. That's what practical edge intelligence should look like.

The traditional model of "one cloud API call per decision" is both expensive and latency-constrained at high frequencies. When classification is reduced to a lightweight "one encode + one dot product" operation running on-device, the entire cost and latency structure is fundamentally transformed.

Summary

Decide Live is built by VIDRAFT / FINAL-Bench and open-sourced under the Apache-2.0 license. Its value lies not only in being a playable demo, but in clearly demonstrating a key insight: classification decisions don't need to go through the generation pathway.

Replacing per-token generation with embedding similarity and calibrated softmax preserves determinism while minimizing computational cost. Bringing that to the browser via WebGPU and transformers.js offers a compelling engineering paradigm for edge AI applications that demand high frequency, low latency, and strong privacy guarantees. Interested developers can visit the Hugging Face Space directly to try it out, and inspect the page source to understand the full decision pipeline.