GatherBlockQuantized
GatherBlockQuantized is a contributed operator in ONNX Runtime that fuses table lookup (Gather) with block quantization (Block Quantized). It is designed for the embedding layers of low-bit quantized models, enabling efficient loading of quantized weights.
Timeline (last 90 days)
The asyncify build of onnxruntime-web does not include the CPU kernel for the GatherBlockQuantized operator, so it cannot load and use 4-bit Gemma ONNX models with block-quantized embeddings on pure WebAssembly (no WebGPU) devices.
The regular build of onnxruntime-web (ort-wasm-simd-threaded.{mjs,wasm}) includes the CPU kernel for GatherBlockQuantized and can support running 4-bit Gemma on WebAssembly.
GatherBlockQuantized is a contrib operator of ONNX Runtime that fuses Gather (table lookup) and Block Quantized, designed specifically for low-bit quantized embedding layers, and is not a standard ONNX operator.
okasi/onnx-lab#2 documented the problem of asyncify lacking GatherBlockQuantized and switched to the JSEP build to work around it; dephelion/ufeed#64 also encountered a similar problem.
All Facts (4)
The asyncify build of onnxruntime-web does not include the CPU kernel for the GatherBlockQuantized operator, so it cannot load and use 4-bit Gemma ONNX models with block-quantized embeddings on pure WebAssembly (no WebGPU) devices.
50%UnverifiedThe regular build of onnxruntime-web (ort-wasm-simd-threaded.{mjs,wasm}) includes the CPU kernel for GatherBlockQuantized and can support running 4-bit Gemma on WebAssembly.
50%UnverifiedGatherBlockQuantized is a contrib operator of ONNX Runtime that fuses Gather (table lookup) and Block Quantized, designed specifically for low-bit quantized embedding layers, and is not a standard ONNX operator.
50%Unverifiedokasi/onnx-lab#2 documented the problem of asyncify lacking GatherBlockQuantized and switched to the JSEP build to work around it; dephelion/ufeed#64 also encountered a similar problem.
50%