Why Real-Time AI Inference Costs 56x More Than It Should — And How Persistent Kernels Fix It

The Cost Crisis in Real-Time AI Models

A team building an inference engine (dotwave.ai) recently published two technical notes targeting a problem the industry has largely overlooked: serving real-time AI models today may cost up to 56x more than it should. They used NVIDIA's full-duplex voice model, Nemotron VoiceChat 11B, as their test case to dissect both the root cause and the solution.

The central argument is sharp: given the same model weights, the same precision, and the same output, the cost gap comes entirely from how the model runs — not from the hardware or the model itself.

Request-Based Inference vs. Continuous Inference

Understanding this problem starts with distinguishing two fundamentally different inference paradigms.

Traditional "Request-Based" Inference

The vast majority of AI inference today is organized around discrete requests: input arrives → model runs → response ends → resources are released. If the server is busy, it can wait briefly to batch multiple requests together, or reorder the queue — at the cost of a slightly slower response. This logic is the design foundation of mainstream inference serving frameworks like vLLM, SGLang, TensorRT-LLM, and Triton.

The Rise of "Continuous Inference"

But there's a class of models that works completely differently. These models remain active in parallel with some external entity — a conversation, a video stream, a robot — continuously receiving input and maintaining state throughout the entire session. The team calls this "continuous inference."

Full-duplex voice is the clearest example. The model listens and speaks simultaneously, so you can interrupt it at any time. Audio arrives continuously throughout the call, conversational memory must stay alive until the call ends, and the next output frame must be delivered every 80 milliseconds. That deadline arrives whether the server is ready or not — if a frame is late, you hear a glitch. What makes this even trickier is that the model must also run during silence: the length of a pause is precisely how the model determines whether you're just hesitating or whether you've finished speaking.

This is fundamentally different from traditional voice agents (speech-to-text → LLM → text-to-speech). In that pipeline, the LLM is idle between conversation turns. Continuous inference has no such gaps to skip.

A note on full-duplex vs. half-duplex: The core difference lies in the simultaneity of communication. Half-duplex systems (like walkie-talkies) can only transmit in one direction at a time — traditional voice agents follow this model, where one side "finishes" before the other begins. Full-duplex systems allow both ends to send and receive simultaneously, like a regular phone call. For AI, full-duplex means the model must process incoming audio in real time while continuously generating speech output, ready to respond to interruptions at any moment. This requires maintaining a dynamic state (KV Cache) spanning the entire conversation, rather than clearing context after each complete response. NVIDIA's Nemotron VoiceChat 11B is specifically designed for this paradigm, and its 11B parameter scale means that even a single session carries a significant VRAM footprint — amplifying the cost penalty of inefficient scheduling.

Why It's So Expensive

Existing inference frameworks are built around requests, and several key assumptions collectively break down in continuous inference scenarios:

• Batches can't wait. At every tick, the server must run all due tasks with whatever resources are on hand — there's no buffer.
• Overload hits everyone simultaneously. Sessions batched together share the same compute step; one slow step delays the entire group.
• Memory can't be freed or swapped mid-session. Because it will be needed again in 80 milliseconds.
• Standard metrics mask failures. Tokens per second and average latency may look fine while a caller has been hearing glitches the entire time.

The team published a striking set of numbers: in NVIDIA's reference stack, each 160-millisecond audio chunk is broken into thousands of tiny GPU operations, with the CPU coordinating between them — leaving the GPU idle roughly 75% of the time. A single session barely manages to run. Add a second session, and 8–15% of audio ticks start missing their deadlines.

In other words, every real-time conversation monopolizes an entire H100, and that expensive GPU sits idle most of the time.

The Fix: Pin the Work to the GPU

The team emphasizes that the fix doesn't require faster GPUs or a different model — it requires rethinking how the model runs. Three core principles drive their approach:

Keep the Work Resident on the GPU

Their engine runs the entire model as a single persistent GPU program. At each tick, the CPU only feeds in new audio and retrieves output — nothing else moves between CPU and GPU. This directly eliminates the 75% idle time seen in the reference stack.

What's happening under the hood: This corresponds to "kernel fusion" and "persistent kernel" techniques in GPU computing. In traditional inference, each GPU operation (matrix multiplication, activation functions, attention computation, etc.) is launched as an independent CUDA kernel, requiring the CPU to schedule and move data between each launch — generating significant launch overhead and PCIe bandwidth consumption. The persistent GPU program approach compiles the entire forward pass into a single continuously running kernel: GPU threads don't exit or reinitialize between audio ticks; they simply wait for the next batch of input data. This compresses CPU-GPU interactions from thousands of per-step operations down to a single boundary sync per tick. This design philosophy is similar to CUDA Graphs, but with more aggressive static compilation tailored for latency-critical streaming scenarios.

Advance All Active Sessions in Lockstep

All sessions due at the same tick are run as a single batch, so model weights are loaded from memory once for the entire group rather than once per session. This is the key lever for amortizing cost.

Decide Everything Ahead of Time

Since the work performed at each tick is identical every time, a compiler can fix the scheduling and memory layout before the first call ever arrives. At runtime, there's no scheduler or memory allocator making decisions on the fly — and therefore no additional latency introduced by those decisions.

What's happening under the hood: This is essentially Ahead-of-Time (AOT) compilation applied at the inference scheduling layer. Mainstream inference frameworks typically use dynamic scheduling: at each request, the runtime decides memory allocation and operator ordering based on current VRAM availability, batch size, and sequence length — flexible, but prone to unpredictable latency jitter. When the workload structure is fully fixed (one frame every 80ms, fixed batch size, fixed context length ceiling), the scheduler can "burn in" all decisions into a static execution graph before the first call: memory addresses are fixed at compile time, operator ordering is determined, GPU thread allocation requires no runtime negotiation. This eliminates scheduling overhead and is the underlying reason why p99 latency stays stably within 147.5ms with zero missed deadlines across 84,000 measurements — deterministic systems are inherently better suited to hard real-time constraints than dynamic ones.

Benchmark Results: From 1 to 56

On a single NVIDIA H100 SXM 80GB, the team measured:

• 56 concurrent sessions at peak capacity, versus just 1 in the reference stack;
• p99 latency per 160-millisecond tick of 147.4–147.5 milliseconds;
• Zero missed deadlines across three rounds and 84,000 session tick measurements.

The cost per conversation equals GPU time divided by the number of conversations sharing that GPU. Going from 1 to 56 means each conversation consumes roughly 98% less GPU time.

The team is also transparent about measurement boundaries: these are server-side figures, excluding network and audio playback, using identical recorded inputs with approximately two minutes of context.

Implications for Real-Time AI Deployment

This technical note poses a question worth asking for every team building real-time models: How many concurrent sessions can you run per GPU today — and how do you verify that none of them are missing deadlines?

As full-duplex voice, embodied intelligence, real-time video understanding, and other "continuous inference" scenarios grow rapidly, inference stacks built around the request paradigm are exposing structural weaknesses. This case points to a broader trend: the next frontier in inference optimization may not be model compression or more powerful chips — it may be the scheduling paradigm itself.

When there's a 56x gap in hardware utilization, architectural innovation at the software layer is the real cost lever.

Note: The data and conclusions above are drawn solely from technical notes published by dotwave.ai and represent a single source. Independent third-party replication has not yet been conducted.