Building a Small Multilingual Offline Voice-Control Pipeline with ONNX

Many products don't need a full speech-to-text system to support voice control. A smart switch might only need a wake word and a handful of fixed commands like "turn on the light," "turn off the light," "brighter," and "dimmer." For this fixed-vocabulary scenario, keyword spotting (KWS) is a better fit than running a general-purpose ASR model all the time. This article covers the pipeline design for small offline keyword models, the boundaries of what KWS can do, and how the same model can move from browser-based testing to an embedded product.

The Fundamental Difference Between KWS and ASR

ASR answers the question "What did the user say?" while KWS answers "Did the phrase I care about appear in this audio stream?" The processing paths are completely different:

ASR:
audio → speech-to-text → text matching / intent parsing → action

KWS:
audio → acoustic features → keyword model → command id → action

When the command set is small and predefined, KWS avoids the need for a large transcription model and a second layer of text matching. But it can't replace free-form voice interaction. If users need to say "set the living room light to 50%," the product typically still needs ASR and intent parsing to handle room locations and numeric values.

Mel features (Mel-spectrogram) are the shared first step in both KWS and ASR pipelines. Human hearing doesn't perceive frequency linearly—it has strong resolution at low frequencies and weaker resolution at high frequencies. The Mel scale is a frequency ruler that mimics this perceptual characteristic. In practice, short audio frames (e.g., 25ms window, 10ms hop) are first FFT'd to get a spectrum, then compressed through a bank of triangular Mel filters into 20–40 frequency bands, and finally log-transformed to produce a feature map. Compared to raw waveforms, Mel features reduce the data volume by one to two orders of magnitude while preserving the spectral envelope information most critical for speech recognition—this is precisely what enables small keyword models to run within a few MB of memory. Once the feature parameters (sample rate, frame length, number of Mel bands) are fixed, training and inference must match exactly, or model accuracy will drop noticeably. Additionally, Mel features themselves contain no phase information and are somewhat robust to pitch differences, which is one reason the same features can support multilingual keywords.

Offline Runtime Pipeline Design

In a usable always-listening system, the model is only one part. Raw scores shouldn't automatically trigger product actions—the runtime also needs temporal confirmation, refractory handling, background energy detection, and application-layer state management.

microphone frames
      ↓
audio buffering
      ↓
Mel features
      ↓
keyword model
      ↓
matched keyword outputs
      ↓
consecutive-frame and false-trigger logic
      ↓
application callback

Multi-keyword models can handle the scenario where a product needs a wake word plus several commands. Rather than loading a separate model for each phrase, a single inference pipeline can expose multiple outputs:

"Hey Lamp"       → WAKE
"Turn it on"     → LIGHT_ON
"Turn it off"    → LIGHT_OFF
"Brighter"       → BRIGHTER
"Dimmer"         → DIMMER

This doesn't eliminate the need for testing. Phrases with similar pronunciations can interfere with each other, and each output has a different recall/false-trigger tradeoff.

Temporal confirmation and refractory logic are necessary because keyword models typically output scores frame-by-frame or over a fixed sliding window, and a single frame exceeding the threshold doesn't mean a complete phrase was spoken. A common approach is to require N consecutive frames (e.g., 3–5 frames) to all exceed the threshold before declaring a hit—this is "consecutive-frame confirmation." After a hit, the system enters a refractory period during which the same keyword is ignored for a few hundred milliseconds, preventing a single utterance from being counted multiple times. Background energy detection (e.g., based on RMS or VAD) is used to skip inference entirely during silence or very low SNR, saving power and reducing false triggers. These mechanisms are simple, yet they directly determine the product experience: too high a threshold causes missed detections, too low causes frequent false activations, and the number of consecutive frames and cooldown duration are hyperparameters that need to be tuned per command length. Multi-keyword models must also consider competition between outputs—the arbitration rules for when multiple outputs exceed the threshold in the same frame should also be defined explicitly at runtime.

Why Small Models and Multi-Platform Deployment

The model needs to run continuously while the product waits for commands. Size and compute directly affect RAM and flash requirements, battery life, wake latency, the cost of keeping the main processor active, and whether the model can fit on a microcontroller.

The open-runtime demo model is purpose-built for small deployments. The repository includes ONNX paths for Python, Web, and Android, plus an INT8 TFLite path for the ESP32-S3.

Keep in mind: the final result still depends on the audio frontend and the target device. Desktop results shouldn't be presented as if they were ESP32 results.

INT8 quantization and TFLite are the key techniques that let keyword models run on microcontrollers. INT8 quantization compresses what were originally 32-bit floating-point weights and activations down to 8-bit integers, shrinking model size by roughly 4× while potentially speeding up inference by 2–4× on MCUs that support vector instructions. The cost is a slight accuracy drop, typically mitigated through post-training quantization with a small calibration set (representative dataset). Chips like the ESP32-S3, which feature DSP instructions and vector extensions, have good support for INT8 operators, and TFLite Micro provides a minimal runtime with no OS dependency. The differences between the ONNX path and the TFLite path go beyond quantization—they also include operator sets, post-processing implementations, and inference scheduling—so the same model may produce different score distributions on the two platforms. When porting across platforms, thresholds should be recalibrated rather than reusing desktop parameters, otherwise recall and false activation rates will both deviate from expectations.

Pre-Launch Testing Checklist

"The model triggered once in a demo" is far from sufficient. At minimum, you need to measure:

1. Real-world trigger recall rate
2. False activations per hour
3. Distance and speaking volume
4. Different speakers and accents
5. TV, music, fans, and room noise
6. Model size and device memory
7. End-to-end latency and power impact

Recall and false activation rates must be considered together. Lowering the threshold makes the model trigger more easily but also increases false activations.

Validation Workflow: From Phrases to Device Testing

For teams without a speech ML background, a practical validation workflow is:

1. Define the exact fixed phrases
2. Generate the first model
3. Test with real speakers before downloading
4. Run on the target microphone and device
5. Collect missed detections and false activations
6. Add real speech samples or hard negatives as needed

Voicute is the online training and testing component in this workflow, and the onnx-wakeword repository contains the open offline runtime and platform examples. The key distinction: online testing reduces the risk of downloading the wrong model, but it doesn't replace validation on the final hardware.

Hard negatives are audio clips that resemble the target keyword but shouldn't trigger it—for example, similarly pronounced phrases, homophones, similar words appearing on TV, and background voices other than the target speaker. Ordinary negatives make it easy for the model to learn "just suppress scores for anything that isn't the target word," but hard negatives force the model to focus on genuinely discriminative features, thereby reducing false activations in real environments. A typical way to collect hard negatives is to first deploy a version of the model, record all false-trigger segments, and then retrain or fine-tune with those alongside true trigger segments. You should also collect missed-detection samples and add positive samples across different accents, speaking rates, and distances, so the model isn't only sensitive to a small number of speakers. Hard negatives don't need to be large in number, but their coverage should closely match real sources of false triggers—this is often more effective than simply adding more positive samples.

Deciding Which Architecture Fits

Use a small KWS pipeline when: the vocabulary is fixed, the device needs to run locally or offline, the device is battery-powered or resource-constrained, low latency matters, and the product doesn't want to send continuous audio to a cloud service.

Use ASR and intent parsing when: users can express the same intent in many ways, commands include open values/numbers/entities, and the product is closer to a conversational interface.

Disclosure: I'm the creator of Voicute. The model generation service is commercial, while the offline inference runtime and integration examples are open source. The claims above should be verified on your own microphone, language, and target device.

One-line Summary

A small offline KWS pipeline brings fixed-vocabulary voice control to low-power devices using ONNX/TFLite.

Paragraph Summary

This article discusses how to build a multilingual offline voice-control pipeline using small keyword spotting (KWS) models instead of running full ASR. KWS only determines whether a predefined phrase appears in an audio stream, with a path of audio → Mel features → keyword model → command ID, making it suitable for fixed-vocabulary, low-latency, resource-constrained scenarios. A complete pipeline also requires audio buffering, temporal confirmation, refractory logic, and background energy detection. Multi-keyword models can expose wake-word and multiple command outputs in a single inference pass, but similarly pronounced phrases can interfere with each other. The model supports ONNX paths for Python/Web/Android and an INT8 TFLite path for ESP32-S3, but desktop results don't represent MCU performance. Before release, you must measure recall, false activations per hour, distance and volume, accents and noise, memory, and power. Online testing doesn't replace final hardware validation.