Offline Voice Control with ONNX: Custom Wake Words and Multi-Command Recognition

Title

Offline Voice Control with ONNX: Custom Wake Words and Multi-Command Recognition

Meta Description

Learn how to build offline voice control with custom wake words and ONNX: a two-stage architecture separating wake and command recognition, a multi-keyword model for fixed commands, and R0/R1 two-round training to govern false triggers, plus ONNX and TFLite deployment checklists.

Article Content

Many voice-controlled products don't need to transcribe every sentence. When interaction starts with a wake word followed by only a small set of fixed commands, on-device keyword spotting is often simpler and more controllable than continuously running cloud speech recognition. Based on an open-source implementation, this article breaks down how to build offline voice control with custom wake words plus ONNX inference, focusing on two easily underestimated areas: false-trigger governance and cross-runtime deployment.

Two-Stage Architecture: Separating Wake and Command

The core of this approach is splitting "wake" and "command recognition" into two stages. The audio stream captured by the microphone first goes into the wake word model, which runs continuously with a fairly conservative threshold; once the wake word is hit, the app opens a short time window, passes the audio within that window to a multi-keyword model for command recognition, and finally maps it to a concrete action.

Take the example from the original article: the wake word is "Hey device" and the command is "Turn on the light." The value of this design is that the larger command recognizer doesn't need to run all the time, and the interaction state machine is easier to control—the system clearly knows whether it's currently in the "waiting for wake" or "receiving command" phase.

Why Not Use Speech Recognition for Wake Words

The Web Speech API is convenient during prototyping, but it is not a reliable always-on wake word detector. The original article points out that browser support for SpeechRecognition and its continuous mode is inconsistent; some implementations may drop the connection midway or actually send audio to a remote recognition service—which introduces latency and breaks the privacy premise of "audio never leaves the device."

A more robust local path is: request microphone permission after a user gesture, read the audio stream with getUserMedia(), convert the stream into the frame format the model expects, run keyword spotting inference locally, and only start ASR or show the command interface after the wake word is detected. The browser still has permission and lifecycle constraints, but the keyword spotter no longer depends on SpeechRecognition returning transcription text.

The Web Speech API is a high-level speech interface provided by browsers, with SpeechRecognition as its core object. It is designed for "speech-to-text," not "always-on keyword spotting," and therefore has several structural limitations: first, continuous mode support is uneven outside Chrome, and some implementations automatically end the session after a few seconds of silence, requiring repeated restarts; second, most browsers send audio to a cloud recognition service (such as Google's servers), meaning wake word detection latency depends on network round-trips and audio leaves the device; third, the API does not expose raw audio frames or confidence scores, so applications cannot customize thresholds or perform local framing. These limitations make it suitable for one-off dictation scenarios but not for wake word scenarios that require 24/7 low-power listening. After switching to getUserMedia() to obtain a MediaStream, audio data stays local, and developers can control the sample rate, frame length, and inference cadence themselves.

One Model for Multiple Fixed Commands

When a device only needs to handle a limited set of commands, a multi-keyword model can output a score for each label, and the app maps the selected label to an action. The command set given in the original article includes next, previous, stop, open menu, and close menu, with a code example that uses an actions object for dispatch:

const actions = {
  next: () => nextPage(),
  previous: () => previousPage(),
  stop: () => stopPlayback(),
  "open menu": () => openMenu(),
  "close menu": () => closeMenu(),
};

function handleKeyword(label) {
  actions[label]?.();
}

The boundaries of this approach need to be stated clearly: it applies to scenarios where the vocabulary is known in advance and cannot replace open-ended speech recognition. Utterances like "set the living room light to 37%" that contain arbitrary parameters still require ASR plus intent parsing, because the vocabulary and values are not fixed.

The difference between a multi-keyword model and a single wake word model lies in the output layer: the latter usually outputs only one "target word vs. non-target word" score, while the former outputs an independent score for each keyword label (a common approach is sigmoid multi-label output rather than softmax single-label mutual exclusion). This means the same audio frame can theoretically activate multiple labels at once, and the app needs thresholds and cooldown logic to disambiguate. During training, each keyword needs positive samples, while all non-target audio (including samples of other keywords) serves as negative samples, so the model learns to distinguish between labels. Vocabulary size affects model capacity and false-trigger rate: the more labels, the higher the confusion risk, so it is generally recommended to keep the number of commands within a dozen or so and perform additional hard negative mining for similarly pronounced words.

False-Trigger Governance: R0 and R1 Two-Round Training

Getting the model to recognize the target phrase is only half the problem; real devices must also learn to reject: everyday conversation, music and TV sound, fans and mechanical noise, similarly pronounced phrases, and noise introduced by the microphone or enclosure.

The original article breaks this process down clearly: the first model version can be regarded as R0; after testing, collect the audio clips that were falsely triggered, and these "hard negatives" go into the next round of training to produce R1. R1 is a training process, not an additional model that must run at runtime—the goal is to teach the model to reject easily confused sounds while maintaining recall of the target phrase. This is often misunderstood as "deploying two models," but in reality there is still only one model at runtime.

Hard negatives are the application of the "hard example mining" idea in machine learning to keyword spotting. Ordinary negative samples are randomly collected background audio that the model easily classifies as negative; hard negatives are clips the model currently misclassifies as positive—for example, similar syllables on TV, a user coughing, or a door closing. Adding these clips to the training set and retraining is essentially a local correction targeted at the model's decision boundary, and it usually significantly reduces the false-trigger rate without substantially sacrificing recall. In practice, after the R0 model is deployed to test devices, you should enable a "false-trigger recording" log, save every clip that exceeds the threshold, manually review and remove genuine target words, and use the rest as training negatives for R1. This process can be iterated over multiple rounds (R2, R3), each targeting the new sources of confusion exposed by the previous round.

Local Inference and Privacy Benefits

On-device keyword spotting suits products that require low latency, offline operation, predictable command behavior, reduced cloud and network dependence, and stronger control over microphone privacy. Note that the model file is only part of the system; the runtime also needs audio capture, framing, feature extraction, threshold handling, cooldown logic, and application callbacks. The original article specifically reminds us to validate the complete pipeline, not just whether the model file can load.

ONNX and TFLite Deployment Essentials

ONNX is suitable for desktop, Python, Android, and browser runtimes that support ONNX Runtime; TFLite or TFLite Micro is common on resource-constrained embedded devices. Before migrating a model to another runtime, verify item by item: sample rate, PCM format, frame length and frame shift, Mel feature configuration, input tensor shape, output label order, quantization settings, and detection threshold.

A key reminder from the original article is that even if the model file is completely unchanged, recall can drop as soon as preprocessing changes. This explains why many cases of "the model is fine but performance got worse" actually have their root cause in feature extraction.

Mel features (Mel spectrograms) are the most common input representation for keyword spotting models. They simulate the human ear's nonlinear perception of frequency: the linear spectrum is mapped to the Mel scale through a set of triangular filter banks, then the logarithm is taken. Key parameters include sample rate (commonly 16 kHz), frame length (e.g., 25 ms), frame shift (e.g., 10 ms), number of Mel filters (e.g., 40 or 64), frequency upper and lower bounds, and whether normalization is applied. These parameters must be exactly consistent with training; otherwise, the feature distribution extracted on the deployment side for the same audio will shift, the input the model sees will not match the training distribution, and recall will drop accordingly. When migrating across runtimes, especially check whether TFLite's Microfrontend or custom operators align with the Python-side librosa/torchaudio implementation, including the window function type (Hann or Hamming) and whether zero-padding is applied.

Practical Test Checklist: Don't Optimize Only for Recall

The test checklist given in the original article is worth using directly: test the wake word with multiple speakers; test different distances and speaking rates; play ordinary conversation and background audio; count false triggers per hour; measure recall and false-trigger rate separately; add hard negatives for the next round of training; test cooldown and repeated-trigger behavior; and confirm that every recognized command maps to the expected safe action.

The most easily overlooked item is: don't optimize only for recall. A model that triggers too often is unusable at the product level even if it rarely misses the target phrase. False triggers are the real killer of the voice-control product experience.

Open-Source Implementation and Further Reading

The open-source project onnx-wakeword provides inference code and examples for running ONNX wake word and keyword spotting models in local applications, making it suitable for validating the inference path before integrating into a larger product. Of course, the final result should still be tested with real microphones, the target operating system, and hardware. The original article also provides a longer architecture guide and a German-language voice control demo video for reference.

One-line Summary

Build offline voice control with custom wake words and ONNX local inference, with a focus on governing false triggers and cross-runtime deployment.

Paragraph Summary

The article breaks down an offline voice control solution based on an open-source implementation: wake word detection and command recognition are split into a two-stage architecture, with the wake model running continuously and a short window opened only after a hit, passed to a multi-keyword model to recognize fixed commands. It explains why not to rely on the Web Speech API for wake words, emphasizing its unstable connections and the privacy issue of audio being sent off-device; it introduces using hard negatives for R0-to-R1 two-round training to govern false triggers, and clarifies that only one model is still needed at runtime. The deployment section compares the applicable scenarios of ONNX and TFLite, reminding that sample rate, frame parameters, Mel configuration, and label order must be checked item by item, since changes in preprocessing directly lower recall. Finally, it provides a practical test checklist, emphasizing that recall and false triggers per hour should be measured separately.