Fixing GatherBlockQuantized: Running 4-bit Gemma in Browsers Without WebGPU

Running a 4-bit quantized Gemma model in the browser is an appealing prospect: no GPU required, just pure CPU letting a small model answer questions locally. But if you're using Transformers.js 4.x to load a 4-bit Gemma ONNX model on a WebAssembly device (no WebGPU), you'll likely run headfirst into this remarkably calm error:

Can't create a session. ERROR_CODE: 9, ERROR_MESSAGE: Could not find an
implementation for GatherBlockQuantized(1) node with name
'/model/embed_tokens/Gather_Quant'

The good news is that the model itself is fine — the problem is that the wrong runtime file was selected. Pointing ONNX Runtime at the regular WebAssembly build bypasses this pitfall entirely.

Root Cause: The Default Build Is Missing an Operator Kernel

Gemma's 4-bit ONNX export makes the embedding table block-quantized, so the first node in the computation graph, /model/embed_tokens/Gather_Quant, is a GatherBlockQuantized operator. This is a contrib operator in ONNX Runtime, and not every runtime build ships with its CPU kernel.

onnxruntime-web publishes multiple WebAssembly builds in its dist/ directory. Transformers.js 4.3.1 defaults to the asyncify build (ort-wasm-simd-threaded.asyncify.{mjs,wasm}), which is designed for the newer WebGPU execution provider. It doesn't include the CPU kernel for GatherBlockQuantized, so session creation fails on the very first node.

The regular build (ort-wasm-simd-threaded.{mjs,wasm}), on the other hand, does include this kernel. Transformers.js 4.3.1 (paired with onnxruntime-web 1.31.0-dev.20260914) only automatically uses the regular build in one case: Safari below version 26 without WebGPU. In other words, Chrome users get routed down the asyncify path — and hit the wall.

The key takeaway is simple: you don't need to swap model files, you don't need a GPU, and you don't need to change the quantization method. You just need to swap one runtime file.

What is a contrib operator? ONNX defines a standard set of operators (MatMul, Gather, Softmax, etc.) that any compliant runtime should theoretically implement. But the community and vendors keep proposing new operator requirements beyond the standard set, and these are gathered under ONNX Runtime's contrib namespace, hence "contrib operators." GatherBlockQuantized is one of them: it fuses "table lookup (Gather)" and "block quantization (Block Quantized)" into a single operator, purpose-built for the embedding layers of low-bit quantized models. Because it's an extension operator rather than a standard one, not every ONNX Runtime build compiles its CPU kernel — which explains why the same model file behaves so differently across runtime builds.

What is the asyncify build? Asyncify is a compilation technique provided by Emscripten that allows synchronous C/C++ code to "pause," wait for an asynchronous operation to complete, and then resume execution. This is critical in WebGPU scenarios, because GPU command submission, buffer mapping, and similar operations are fundamentally asynchronous. onnxruntime-web's asyncify build is optimized precisely for this, but the trade-off is that it trims some CPU-side operator implementations to keep size and complexity under control. The regular build, conversely, aims for full CPU operator coverage but lacks asyncify's asynchronous suspension capability. Choosing between them is essentially a trade-off between "WebGPU support" and "complete CPU operators."

The Fix: Manually Specify wasmPaths

The solution is to point ONNX Runtime's wasmPaths at the regular WebAssembly build before any model is loaded. The following code must run before the first pipeline() call, because the runtime only starts up once per page:

import { pipeline, env } from 'https://cdn.jsdelivr.net/npm/@huggingface/transformers@4.3.1';

// Must run before the first pipeline() call: the runtime only starts once per page.
const ort = env.backends.onnx;
const dist = `https://cdn.jsdelivr.net/npm/onnxruntime-web@${ort.versions.web}/dist/`;
ort.wasm.wasmPaths = {
  mjs: `${dist}ort-wasm-simd-threaded.mjs`,
  wasm: `${dist}ort-wasm-simd-threaded.wasm`,
};

const gemma = await pipeline('text-generation', 'onnx-community/gemma-3-270m-it-ONNX', {
  dtype: 'q4',
  device: 'wasm',
});

With this change in place, the 4-bit version of Gemma 3 270M (~323 MB) loads and runs on CPU in Chrome. This fix was produced by an AI agent and only written up after being verified in a real Chrome browser.

How the wasmPaths override works: When onnxruntime-web initializes, it automatically infers which .mjs (JavaScript glue code) and .wasm (WebAssembly binary) files to load based on the runtime environment. Transformers.js layers an additional selection logic on top, deciding whether to point at the asyncify or regular build based on device capabilities (whether WebGPU is available, browser type, and version). ort.wasm.wasmPaths lets you bypass that automatic selection and directly tell the runtime where to fetch files. It accepts an object specifying the URLs for .mjs and .wasm separately. Because this configuration is read once at the runtime's initial startup and then locked into memory, it must be set before the first pipeline() call.

Why use versions.web instead of hardcoding a version number: env.backends.onnx.versions.web exposes the onnxruntime-web version that the current Transformers.js actually bundles and depends on. The .wasm binary and .mjs glue code communicate through an internal ABI contract, and if their versions don't match, you can get hard-to-diagnose runtime errors (function table mismatches, misaligned memory layouts, and so on). Building the URL from versions.web guarantees that the wasm file you download is strictly paired with the loaded JS code, avoiding hidden incompatibilities introduced by manual version upgrades or downgrades.

Two Easy-to-Miss Details

The fix itself is just a few lines of code, but there are two places where a small slip will make it fail.

First, wasmPaths must be set before any model is loaded. The runtime only starts once per page, so if your page loads another model first (say, a BERT NER model before Gemma), that model will lock in the runtime build for the entire page. In the real project, this override code was placed at the top of the module that every page imports first.

Second, watch the version binding. The URL above uses env.backends.onnx.versions.web, so the .wasm file always matches the version of the JavaScript glue code bundled with Transformers.js. Errors caused by manually mixing versions tend to be much harder to diagnose.

It's also worth keeping a fallback: if the 4-bit file still fails to start, you can switch to dtype: 'q8' (~545 MB) and show users which one actually loaded. In browser testing, the 4-bit file loaded successfully on the first attempt.

How This Problem Surfaced

The issue was discovered by an app called Paste-Safe, which redacts client names locally in the browser before you paste text into a chat AI. Gemma 3 270M handles an optional second pass for names that the first model missed.

It's worth noting how it surfaced: all unit tests passed, because unit tests don't load real models. The error first appeared when another agent clicked "Run the test" in a real Chrome window. This says something about how scenarios involving real model loading and runtime selection are hard to cover with unit tests alone. After the change, the same page successfully loaded 4-bit Gemma and completed a 30-message test set.

Is This Problem Known?

Partially. Other developers have hit the same wall in their own projects and worked around it at the code level: okasi/onnx-lab#2 documents "asyncify missing GatherBlockQuantized" and switches to the JSEP build; dephelion/ufeed#64 bundles a JSEP runtime for Gemma 2 q4; Alarm2024/outside-35#1 is a similar case.

But at the time of writing, neither the Transformers.js nor the ONNX Runtime issue tracker contained an entry for this problem, nor any written fix documentation. So the purpose of this write-up is to provide a searchable version. If you happen to be a maintainer of either project and have a better fix (changing the default behavior, or providing a clearer error message), it would help everyone trying to run small Gemma models in GPU-less environments.

From a broader perspective, this kind of problem reflects a reality of on-device AI in the browser: there's implicit coupling among model files, quantization formats, and runtime builds, and the default choices aren't necessarily suited to every device combination. 4-bit quantization compresses the model to 323 MB for easy distribution, but it also introduces dependencies on contrib operators like GatherBlockQuantized, pushing portability problems from the model layer down to the runtime layer. For developers, one rule of thumb is worth remembering: when an ONNX session fails on the very first node and a contrib operator name shows up in the error, suspect the runtime build first — not the model itself.