Why We Built Voice AI for the Device, Not the Cloud

When adding voice features to an app, the default path is to send audio to a server. The microphone captures sound, it's packaged into WAV or Opus, streamed to an API endpoint, and a JSON response with recognized text comes back, after which business logic operates on the text. This pattern dominates not because it's the best solution, but because historically it was the only one that could run at the scale of real production models — open ASR models were too large, on-device models lagged noticeably behind cloud quality, and mobile hardware couldn't run ML inference continuously in the background without draining the battery within an hour.

The VoxRT team built for the device from day one. We didn't migrate from the cloud — we chose a different path from the start.

The economics that changed the team's thinking

Suppose a voice notes app has 100,000 monthly active users, each dictating 20 minutes per month on average. That's 2 million minutes of audio to transcribe. At a major cloud ASR provider's lower-tier rate of about $0.004/minute: 2M × 0.004 = $8,000 per month, $96,000 per year — just for transcription, not including wake words, keyword spotting, or TTS.

The on-device logic is completely different. The engineering investment is upfront and doesn't scale with user growth. Integrating an on-device SDK typically takes 2 to 6 weeks (starting with a single platform), longer for cross-platform or when custom optimization is needed. Ongoing costs exist too: SDK upgrades after model version updates, debugging device-specific regressions, validating new OS versions. But none of these grow linearly with user minutes, and the unit variable cost approaches zero.

One point shouldn't be glossed over: on-device SDKs are not universally free. The VoxRT stack uses a hybrid license (open wrapper layer, proprietary runtime), while other on-device voice vendors use per-seat, per-deployment, or revenue-share pricing. For most teams, engineering cost plus licensing fees together still beat per-minute cloud pricing at scale, but a complete judgment requires counting both sides.

The break-even point depends on the user base. For the hypothetical app above, a few thousand paying users is enough for on-device to win economically. If the app is expected to reach hundreds of thousands or millions of users, on-device almost invariably wins in the long run.

This calculation also ignores second-order effects: no network cold-start latency, no throttling under peak load, no scaling costs as you grow, and no backend team needed to monitor ASR pipeline availability. For a voice notes app running at scale, the first two matter most; for an occasional voice feature in a large app, the engineering commitment of a mobile SDK may not be worth it even if it's ultimately cheaper on a per-minute basis.

RTF (Real-Time Factor) is the most commonly used performance metric in on-device voice inference, defined as the ratio of the time required to process audio to the duration of the audio itself. RTF = 1.0 means processing 1 second of audio takes 1 second, just achieving real-time; RTF < 1 means faster than real-time, with headroom; RTF > 1 means inference can't keep up with audio arrival, causing stutter or dropped frames. The "0.97% RTF" mentioned in the article is actually 0.0097, meaning processing 1 second of audio takes only about 9.7 milliseconds — a magnitude that's already sufficiently instant for in-browser voice UX.

RTF measurement is highly dependent on preconditions: device model, CPU cores and frequency, whether SIMD instructions are enabled, thread count, batching vs. streaming chunks, audio sample rate and bit depth. A single RTF number alone can therefore easily mislead integrators. Responsible publishing means providing a multi-device matrix, clearly specifying the test set and hardware configuration, and distinguishing average RTF from p99 chunk latency — the latter determines worst-case stutter experience.

Three scenarios where on-device wins cleanly

Privacy is a legal requirement. Medical apps with voice notes, legal tools that record client meetings, personal journals. In these scenarios, sending audio to a third-party server isn't a performance issue, it's a compliance issue. When audio never physically leaves the device, operating under HIPAA, GDPR, and the EU AI Act is simpler. Proving to a lawyer that user voice was never sent to a server is far easier when the architecture cannot initiate a network call at all.

The app must work offline. Field tools for inspectors, construction workers, surveyors; travel apps on planes or in countries without data roaming; workflows for healthcare and first responders in remote areas. If users need voice input in these scenarios, pure cloud isn't an option. You can build an offline fallback via caching, but if the on-device component is already doing the work, why maintain a parallel cloud path?

Latency must be under 100 milliseconds. For wake words and hotword detection, the "user spoke, app responded" loop must feel instant. Any network round-trip breaks the experience. Even on perfect 5G, RTT alone costs 50 to 150 milliseconds, plus server-side inference. On-device wake word inference typically runs between 50 and 200 milliseconds end-to-end. For the wake detection flow, this is the difference between "usable" and "annoying."

These three categories aren't edge cases — they're a substantial portion of real voice use cases: privacy-sensitive apps, offline-first tools, and the always-listening trigger layer within larger voice stacks.

RTT (Round-Trip Time) refers to the total time from when a packet leaves the client to when the server response is received, comprising the outbound leg, server-side processing, and the return leg. On mobile networks, RTT is typically far higher than theoretical bandwidth would suggest: even under ideal 5G, physical propagation, base station scheduling, and protocol stack overhead add tens to over a hundred milliseconds.

For always-listening trigger layers like wake words, the user's perception threshold is very sensitive: beyond about 100 milliseconds it noticeably feels "a beat behind," and beyond 200 milliseconds it approaches being perceived as laggy. Cloud solutions must additionally layer on RTT plus server-side queueing and inference time, whereas on-device wake word inference completes on the same device, eliminating the uncontrollable network round-trip. This is also why low-latency scenarios often aren't a case of "the cloud isn't optimized well enough," but of architecture that simply cannot compete with on-device.

Where on-device is genuinely hard

This article wouldn't be honest without this section.

Model size vs. quality trade-offs. To fit a model into 100 KB (VoxRT's wake word), the architecture must be deeply pruned. The wake word accepts a fixed trigger phrase ("Hey Assistant") and ships only one validated model, rather than a custom-trained one per user. This works for the scenario but means custom trigger phrases can't be offered without exposing the training pipeline. Streaming ASR models are larger (medium models around 60 MB of weights), still far below the multi-GB models cloud providers run server-side.

Hardware fragmentation is real. The stack covers iOS, Android, browsers (via WASM), and Linux (Raspberry Pi, Jetson, servers). The shared Rust runtime compiles to each target, but each target still requires its own benchmark runs and build tuning for local SIMD extensions, memory layout, and audio API conventions. Publishing a single "RTF 0.05" number across all platforms would be dishonest. The reality is 5 to 10 benchmark points on different devices, each with its own preconditions. This is more work than cloud providers face, because their infrastructure is uniform while on-device is not.

Updates go through the SDK release cycle. If cloud ASR has a bug, push the fix to the server and everyone gets it tomorrow. On-device, a bug means a new SDK version, an npm/pip/CocoaPods update, and users who must upgrade. Mobile developers are familiar with this pain from any native library. A stable C ABI between the runtime and SDK bindings can reduce the frequency of breaking changes, but it can't eliminate the user upgrade step. This is unresolved friction.

Benchmarking requires discipline. Picking one device to publish a pretty number is easy. Publishing a multi-device matrix with measured RTF, different CPU budgets on different cores, and p99 chunk latency is hard. VoxRT deliberately makes it harder on itself, because the alternative misleads integrators into building on expectations that don't match their hardware. With every new flagship phone launch, the problem compounds.

On-device debugging in production is thinner. When cloud ASR regresses for a specific user, the server team has logs, metrics, and the ability to replay audio. On-device, your telemetry is limited to what you can get back from user devices without violating the privacy premises that led you to choose on-device in the first place. VoxRT only returns RTF counters and detection scores via opt-in telemetry, and even then only to the extent users are willing to share. Diagnosing a regression in a particular chip family often means buying that device and reproducing locally.

SIMD (Single Instruction, Multiple Data) is a class of CPU instruction set extensions that allow one instruction to process multiple data elements simultaneously, delivering several-fold speedups for core operators in voice models such as matrix multiplication and convolution. Common implementations include x86's AVX/AVX2/AVX-512, ARM's NEON, and SIMD128 exposed through WebAssembly in browsers.

SIMD128 is WASM's 128-bit vector extension, supported in Chrome 91+, Firefox 89+, and Safari 16.4+. Before it, in-browser voice inference could only take the scalar path, typically achieving only 2-4x real-time, which couldn't support serious on-device ML. With SIMD128 enabled, the same model can drop below 1% RTF, making browsers a true first-class inference target. This is why the article repeatedly emphasizes "build tuning for local SIMD extensions": different devices and browsers have different levels of SIMD support, and benchmarks must be run separately per platform.

Where these trade-offs ultimately land

The following numbers aren't a pitch — they're evidence: the on-device path has a real engineering form in 2026, not just an ideal.

A wake word model that fits into 100 KB and uses only 5.3% of a single A53 core on a $15 Raspberry Pi Zero 2 W sits at the extreme of today's on-device capabilities. On a Pi 4 B or Pi 5 it drops to 1% to 2% CPU. The constraints that forced the 100 KB target (a single trigger phrase, fixed, no custom training within the SDK) are real limitations, not features. They're published because on such cheap hardware, the alternative is not shipping at all.

A voice activity detector running at 3% CPU on a mid-range Android (Snapdragon 662 class) is the upper bound for keeping always-listening microphone capture from destroying the battery. VoxRT only ships VAD on mobile, not in browsers, because WASM configurations for continuous audio capture vary enough that the engineering investment hasn't yet proven justified.

Keyword spotting running 14 fixed English commands at 0.97% RTF in Chrome WASM and 16% RTF on a Snapdragon 662 makes browser voice UX feel instant. The full browser stack (including runtime) packages to about 1.47 MB — small enough to be irrelevant for most apps, large enough to matter for landing pages optimized for mobile 3G.

Streaming ASR (NeMo FastConformer, 60 MB of weights) runs in real time on mobile. Partial transcriptions arrive while the user is still speaking, finalized after a silence timeout. This model couldn't have shipped on-device three years ago; today it's a shipping choice.

These numbers live in the README at github.com/VoxRT, along with test sets, hardware, methodology, and preconditions. VoxRT doesn't place a subset measurement next to another vendor's full-set measurement. If a wake word is to be compared with other methods, both numbers must be on the same test set — otherwise it's not a comparison.

Where the cloud still wins honestly

The cloud is the right choice for ASR in several scenarios:

Multilingual coverage across 50+ languages. Open ASR models for rare languages are often noticeably worse than cloud. If you need Swahili, Thai, or Tagalog voice, cloud providers still lead. On-device multilingual coverage is catching up, but more slowly.

Rapid model iteration. A cloud team pushes an improved model today, and tomorrow every user gets better accuracy with no app changes. An on-device app must re-release. For products iterating quickly on model quality, this is a meaningful speed difference.

Enterprise compliance tooling around the API. Major cloud ASR providers come with SOC2, HIPAA, and PCI-DSS compliance infrastructure, audit logs, and enterprise SLAs. If you're selling to the Fortune 500, buying compliance through a vendor is sometimes easier than proving it yourself with an on-device deployment.

Open-ended complex workflows. If your voice UX is "ask AI a complex question by voice, run ASR, send to an LLM, get a voice reply," the entire pipeline is already on the server. Adding transcription on the same server costs little extra latency. When every step runs in the cloud, the on-device advantage of the first step is small.

These cases are real. On-device isn't a universal answer — it's the right answer for a specific set of product scenarios, and those scenarios are becoming increasingly common.

GPAI (General-Purpose AI) is a legal classification used in the EU AI Act, referring to foundation models capable of performing multiple tasks and being broadly integrated into different systems, such as large language models and multimodal models. The EU AI Act imposes obligations on GPAI providers including transparency, documentation, copyright policy, and risk assessment, and applies progressively "with enforcement teeth," meaning compliance is no longer a voluntary declaration but may carry actual penalties.

For voice app companies, this regulatory trend turns "whether data leaves the device" from an engineering preference into a legal argument. When the architecture doesn't initiate network calls at all, proving to regulators or customers that user voice never went to the cloud is far more reliable than after-the-fact audit logs and contractual promises. This is the fundamental reason on-device moves from "nice to have" to "compliance posture" in privacy-sensitive segments, and one of the policy drivers behind the article's judgment that on-device share will continue to expand.

Where the trend points

To understand why on-device will capture more market share, it helps to look at how much ground has already shifted. Capabilities that were impractical or research-only three years ago are standard engineering choices today:

Capability	2022-2023 status	2026 status
Streaming ASR on mobile	Whisper-class models (1-3 GB, batch architecture) impractical for real-time mobile streaming	FastConformer-class models (~60 MB of weights) running in real time on ordinary mobile hardware
In-browser voice inference	Pre-SIMD128 WASM, typically 2-4x real-time, Safari didn't support SIMD128 until March 2023	SIMD128 supported in all major browsers, sub-1% RTF achievable
Wake words on cheap hardware	Reproducible open benchmarks on Pi Zero class were scarce, mostly proprietary paid tiers	Open benchmarks published, 5.3% RTF measured on a $15 Pi Zero 2 W
On-device voice cloning TTS	On-device voice cloning at research stage, production voice cloning cloud-only	Shipping products exist, single-step flow sampling makes rapid synthesis a reality

Three trends over a two-to-three-year horizon:

ML capability on mobile chips becomes standard. iPhone's Apple Neural Engine, Google Tensor TPU, Snapdragon Hexagon NPU. Every flagship now has a dedicated chip for ML inference. What was a feat in 2022 (running transformers on a phone) is standard capability in 2026.

Browsers become first-class targets. WASM SIMD128 (available in Chrome 91+, Firefox 89+, Safari 16.4+) opens the door to serious ML inference in browsers. A 100 KB wake word runs at 0.16% RTF on Chrome M4. Impossible a few years ago without a native wrapper. A voice UX entirely within the browser is now a real option.

Privacy becomes a compliance requirement. The EU AI Act begins applying to GPAI models with enforcement teeth. Regulators in the US and Asia are following. For voice app companies, proving to regulators that user voice never went to the cloud is easier when the architecture makes it impossible. On-device goes from nice-to-have to compliance posture.

VoxRT's prediction: over the next 2-3 years, cloud ASR remains the default for broad applications, but on-device captures market share in privacy-sensitive, offline-first, and latency-critical segments. This isn't "on-device will kill the cloud" — it's a reasonable coexistence of two architectural approaches for different use cases.

Repos and resources

The runtime and SDKs are at github.com/VoxRT. Platform-specific bindings:

• Linux (Pi, Jetson, servers): voxrt-wake-word-linux, voxrt-kws-linux, voxrt-asr-linux
• Browser (WASM): voxrt-wake-word-browser, voxrt-kws-browser
• Mobile: iOS and Android bindings are in their respective repos

More on-device voice tech articles at dev.to/voxrtio.

If you've recently evaluated on-device voice but ended up choosing the cloud, what was the deciding factor? Only honest answers are worth hearing.

One-line Summary

On-device voice AI wins in privacy, offline, and low-latency scenarios, but the cloud remains the broad default — the two will coexist long-term.

Paragraph Summary

Using VoxRT's on-device voice stack as an example, the article argues that cloud ASR isn't the optimal architecture but the default path under historical constraints. The economics show a voice notes app with 100K MAU and 20 minutes per user per month costs about $96K/year just for cloud transcription, while on-device has upfront engineering with near-zero unit variable cost and breaks even at a few thousand paying users. On-device wins cleanly in three scenarios: privacy as a legal requirement, must-work-offline, and sub-100ms latency. But on-device has real costs too: model size vs. quality trade-offs, hardware fragmentation, SDK release cycle updates, and limited production telemetry. Measured evidence shows on-device is now engineering-feasible; over the next 2-3 years cloud remains the broad default while on-device keeps gaining share in privacy-sensitive, offline-first, and latency-critical segments.