Browser-Based AI Models Put to the Test: 5 Practical Use Cases That Already Work

How mature is running AI models directly in the browser, really? After the FrontKon conference, one developer ran a systematic round of tests using Transformers.js 4.3.1, and her conclusions turned out to be considerably more positive than she expected.

Why browser-side AI deserves a fresh look

At the FrontKon conference in Prague, developer Sylwia Laskowska gave a talk titled "Your Browser Is a Compute Platform," and was surprised to find that only three or four audience members had ever heard of WebGPU. It made her realize how underappreciated the browser's potential as a compute platform still is among developers.

She had also tested browser-side AI models a year earlier, and her conclusion then was "interesting, but nothing more." But after following the Hugging Face team's progress on Transformers.js, she decided to retest with the latest version before the conference. The result: model quality has improved substantially.

The core advantages of in-browser inference are clear: no calls to external inference APIs, no network round-trip latency, no worries about token consumption costs, and user data stays entirely on the local device. Once a model is downloaded and cached, it can even run offline.

The trade-offs are equally clear: limited model size (you can't run frontier-scale large models), initial download time, and inference speed constrained by device performance.

She integrated five features into a single React + TypeScript app, running inference on background threads via Web Workers so the main UI stays responsive. Models load on demand, WebGPU acceleration is supported, and it falls back to WebAssembly CPU execution when WebGPU isn't available.

WebGPU is a new-generation web graphics and compute API being standardized by the W3C. It lets web pages call the device's GPU directly with near-native efficiency, for both graphics rendering and general-purpose parallel computing. Compared with the earlier WebGL, WebGPU offers explicit resource management, compute shaders, and mapping to modern GPU features (such as Vulkan, Metal, and Direct3D 12), making it particularly well suited to highly parallel workloads like neural network inference. In browser-side AI scenarios, the compute-intensive parts of a model are dispatched to the GPU via WebGPU, which is typically several to dozens of times faster than pure CPU execution. When a device or browser doesn't support WebGPU, Transformers.js falls back to WebAssembly (WASM) running on the CPU—still functional, but noticeably slower.

Transformers.js is a JavaScript library from Hugging Face that aims to run Transformer-family models directly in the browser and in Node.js, with no server required. It executes models through ONNX Runtime Web, supports both WebGPU and WASM backends, and can pull model files on demand from the Hugging Face Hub for caching. Developers can call text, vision, and audio task models with an API similar to Python transformers, greatly lowering the barrier to integrating AI capabilities into purely front-end projects.

Five browser AI features that actually work

Background removal: a 44M-parameter model with surprisingly good results

This feature uses the RMBG-1.4 model, roughly 44 million parameters, with a download size of about 170MB. In testing, it cleanly separated a white shoe from a cluttered background and output a transparent PNG.

The author sees this as a great fit for e-commerce and auction scenarios—a seller uploads a product photo with a messy background, and the platform offers a one-click "remove background" button, with the photo never leaving the user's device.

It's worth noting that RMBG-1.4 has specific licensing conditions, and the scope of authorization must be confirmed before commercial deployment.

Alt text generation: an aid, not a replacement

Using the Florence-2-base-ft vision model (about 230 million parameters, 350–450MB download), it automatically generates descriptions for images.

The author's framing is clear: this is an accessibility aid, not a replacement. Good alt text requires not only recognizing the objects in an image but also understanding the context and purpose for which the image is used. The model doesn't know why you're using a particular image, and occasionally it omits or fabricates details.

She tried having the model incorporate additional context to generate more accurate descriptions, but the results were unsatisfactory. The final implementation simply appended the context text after the generated description, rather than truly getting the model to understand it. She discusses this limitation separately later in the article.

Smart ticket generation: the hardest one to implement

This was the feature the author was most excited about—describe a problem in natural language, and the model automatically generates a structured support ticket.

The model used is Qwen2.5-0.5B-Instruct (about 500 million parameters, roughly 786MB download after quantization).

The author admits this was, unexpectedly, the hardest of the five features. The difficulty wasn't in understanding the complaint, but in getting the model to reliably output correctly formatted JSON without fabricating information or returning malformed structures. It only worked in the end through carefully designed prompts, output parsing, and Zod schema validation.

The generated ticket is editable and only submitted after the user confirms. The author argues that this "AI drafts, human reviews" pattern is the right approach for production environments.

Zod is a schema declaration and validation library for TypeScript. Developers use it to define the structure and type constraints of data, then parse and validate inputs at runtime, producing clear error messages when something is invalid. In scenarios where a language model outputs JSON, Zod often serves as the gatekeeper: the string returned by the model is first parsed into an object, then handed to a Zod schema to check whether fields exist, types are correct, and enum values are valid. That way, even if the model occasionally returns a malformed structure or extra fields, the program can catch it promptly and trigger a retry or fallback, rather than writing dirty data directly into business logic. It was precisely this Zod validation that made ticket generation run reliably in the browser.

Semantic search: where small models truly shine

This is the scenario where the author believes small models genuinely excel. Using the all-MiniLM-L6-v2 embedding model (about 22 million parameters, just 23MB for the quantized CPU version and 90MB for the WebGPU version), it performs semantic retrieval over a collection of support tickets.

The principle is to convert both tickets and queries into vectors and match them by cosine similarity. No generative large model is needed. The model is fast and the results are practical.

From this, the author distilled a key lesson: sometimes a small specialized model is better suited to a specific task than a general-purpose large language model. This is especially true for privacy-sensitive applications—there's no need to send potentially sensitive data to an external server.

With larger data volumes, you need to consider indexing, storage, and performance optimization. Her demo kept the embedding vectors in memory, which was entirely sufficient for the sample size.

An embedding model maps text, images, and other content into fixed-length dense vectors, so that semantically similar content ends up closer together in vector space. all-MiniLM-L6-v2 is just such a lightweight text embedding model, compressing each piece of text into a 384-dimensional vector. The semantic search workflow is: pre-generate and store vectors for all tickets; when a user queries, generate a vector for the query as well; then compute the cosine similarity between the query vector and each ticket vector (the cosine of the angle between the two vectors—the closer to 1, the more aligned in direction and the more semantically related), and return results sorted by similarity. Unlike generative large models, embedding models only "encode" rather than "generate," which makes them small, fast, and a great fit for browser-side and privacy-sensitive scenarios.

Camera pause-and-ask: interesting, but not yet reliable

Using the LFM2.5-VL-450M vision-language model (about 450 million parameters, 350–400MB download), it implements two features: real-time visual description (after turning on the camera, the model periodically describes what it sees), and pausing on a frame to ask questions about the scene.

In testing, the model gave a clearly inaccurate answer to "how old is this woman," which the author herself described as "flattering."

Her assessment: it's worth watching as an experiment in local visual question answering, but for now it can't be treated as a reliable accessibility tool.

Real-world limitations: what's still unsolved

WebGPU coverage and hardware differences

These models perform as well as they do largely thanks to WebGPU. But WebGPU availability and performance vary widely across devices—depending on browser version, operating system, GPU model, and drivers.

The problems are more pronounced on mobile. On the author's Samsung Galaxy S24 FE, two of the five demos couldn't run at all due to timeouts. This isn't an old device, which indicates there are still significant gaps in mobile WebGPU support. Falling back to WebAssembly CPU inference is feasible, but speeds drop noticeably.

Model size is still a barrier

Download sizes range from 23MB for semantic search to 786MB for smart ticket generation. The first-load experience on mobile networks is poor.

Browsers can cache model files, but storage isn't guaranteed to persist. The author's advice: treat these features as optional enhancements, not prerequisites that every user must load.

Some ideas still don't work

The author tried adding image text recognition, experimenting with various models and configurations, but the quality never met demo standards, so she ultimately gave up.

Contextual understanding in alt text generation also never reached an ideal level—she tried using another local language model to fuse visual descriptions with additional information, but no matter how she adjusted it, the results were unsatisfactory.

These failure cases are precisely why you shouldn't blindly trust these models in production.

As for speech recognition, the author has heard that small browser-side models perform well on such tasks, and Transformers.js also supports audio tasks, but she didn't test that this time.

Where browser AI stands today

The author's conclusion is pragmatic and clear: browser-side AI models are already worth exploring in production applications, but not as a replacement for every AI backend, and certainly not as a critical dependency without a fallback plan.

The right positioning is: optional feature enhancements that improve user experience. Specialized small models like background removal and semantic search are already reliable enough; smart ticket generation is usable as long as human review backs it up; and visual question answering and image OCR still need more time.

All the code and an online demo are open-sourced, so interested developers can try it out directly.