Local Voice Assistant on an 8GB GPU: Making the LLM Yield to TTS to Eliminate Stuttering

Running an LLM and TTS simultaneously on a consumer GPU often leads to mid-speech stuttering. While building a local voice assistant on an RTX 4060 Laptop (8GB VRAM), one developer discovered that parallel scheduling, though seemingly efficient, caused the TTS engine to fall badly behind during first-sentence synthesis, draining the playback buffer. His solution was counterintuitive—have the LLM actively pause and wait for TTS, trading generation speed for a smooth voice experience.

Root Cause: Why First-Sentence Synthesis Can't Keep Up

The developer's goal was to enable offline conversation on an ESP32 voice assistant board running the open-source xiaozhi firmware. The system is driven by an RTX 4060 Laptop GPU (8GB) and 16GB of RAM in a laptop: voice detection and recognition run on the CPU, the LLM (Gemma 4 E4B, QAT q4_0 quantized, based on llama.cpp) generates responses, and a voice-cloning TTS (Qwen3-TTS 1.7B) synthesizes speech sentence by sentence.

In isolated testing, the TTS engine needed only about 0.6 seconds to generate 1 second of audio (RTF around 0.6). But once it shared the GPU with the LLM, first-sentence synthesis time inflated to about 1.5x its own length, versus only about 1.0x when the LLM was idle. Generation speed fell behind playback speed, and the buffer was naturally drained.

The median per-sentence RTF for the entire response was essentially the same in both modes (0.84 to 0.89)—the bottleneck was concentrated in the first sentence. The developer hypothesized that TTS frame-by-frame decoding produces a large number of small kernel calls, each queuing behind the LLM's compute queue. Attempts to throttle LLM speed (to 20 then 14 tokens/s) and adjust Windows GPU scheduling priority both failed.

RTF (Real-Time Factor) is the core metric for measuring speech synthesis system speed, defined as the ratio of the time required to generate audio to the duration of the audio itself. An RTF below 1 means synthesis is faster than real-time playback, so in theory audio can be synthesized and played without stuttering; an RTF above 1 means generation can't keep up with playback and the buffer will eventually be exhausted. Note that a per-sentence RTF within spec doesn't guarantee overall smoothness, because the first sentence often bears extra overhead such as model loading, cache warm-up, and compute queue wait times, and its instantaneous RTF can be far higher than the steady-state value. This is precisely the technical root of the phenomenon described in this article: under parallel mode, the "median per-sentence RTF looks normal (0.84-0.89), but the first sentence drags everything down."

Playback Buffer is a key mechanism in streaming audio systems. The sender continuously pushes audio frames to the playback device, and the device maintains a short buffer queue (0.3 seconds in this case). When the buffer queue is emptied and the next frame hasn't arrived, the pipeline "runs dry," manifesting as audible stuttering or pops. A larger buffer tolerates more generation jitter, but at the cost of increased first-word latency—the user waits longer to hear the first sound. So the core tension in streaming voice systems is this: the buffer must be large enough to absorb generation fluctuations, yet small enough to preserve interactive responsiveness. In this article, "first audio latency" and "stutter count" are precisely the two extreme observations of this tension.

Solution: Taking Turns on the GPU

After abandoning parallelism, the developer had the LLM pause after writing each sentence, let TTS synthesize that sentence alone at full speed, then resume writing the next sentence. At a generation speed of 61 to 66 tokens/s, writing a sentence takes far less time than speaking one, so the audio stays ahead of playback.

The cost is that the LLM spends most of its time waiting. In testing, the LLM took about 54 seconds to complete all its writing (instead of 12 seconds), while speech playback lasted 65 to 84 seconds. For a voice scenario, this is perfectly acceptable—the listener is already listening to the response.

Data from Step-by-Step Improvements

Using the same storytelling prompt, the developer sent text at each step (excluding voice detection and recognition) and ran 3 tests each within the same time window. "First audio" refers to the time when the server sends the first audio frame to the board; stutters are estimated by the server based on the board's 0.3-second buffer.

Configuration	Mean First Audio (Range)	Mean Stutters per Response (Count Range)
PyTorch TTS on GPU, LLM on CPU	4.4s (3.5-6.0)	4.59s (48-60)
Native C++ TTS on GPU, LLM on CPU	5.3s (5.2-5.4)	0.05s (0-2)
Both running on GPU simultaneously	3.9s (3.8-4.1)	2.84s (9-17)
Both taking turns on GPU	2.3s (1.9-2.7)	0.09s (0-1)

Switching to the native engine by itself almost eliminated all stuttering, but first-word latency did not improve (5.3s vs 4.4s, with a very noisy baseline). The PyTorch TTS process used about 8.9GB of private memory, most of which was paged to disk on a 16GB laptop, so the first row should not be read as a generic PyTorch-vs-native-engine comparison. Moving the LLM to the GPU made the first word faster but brought stuttering back. Taking turns not only made the first word faster but also eliminated this regression.

Another control test that only changed scheduling used a Japanese prompt, simulated a player with a 0.5-second buffer, and generated identical text under both modes, running 6 times each. In parallel mode all 6 runs stuttered; in turn-taking mode, zero stuttered.

What PCM Tracing Revealed

After posting on the Hugging Face forum, a reader suggested recording the actual arrival time of each audio chunk rather than just looking at averages. The server now logs every sentence sent to TTS, every audio chunk and its duration, and every batch of frames released to the sender.

In parallel mode, TTS continuously falls behind during LLM generation. First audio arrived 2.3 to 2.5 seconds after the request (instead of 1.0 to 1.2 seconds), and across three runs all stutters were concentrated in the first 13 seconds—while the LLM was still writing or had just finished; afterward there were no stutters. On the server timeline, the player needed to wait 1.9 to 3.4 seconds to fully avoid stuttering. In turn-taking mode this figure was 0 to 0.12 seconds. This "how long the player needs to wait" metric is best suited for cross-GPU comparisons.

The last remaining stutter came from the server itself, not the GPU. To trim trailing silence, the server withholds the last 1.2 seconds of each segment. TTS delivers the first 1.0 to 1.2 seconds as a burst, so nothing is sent until about 0.19 seconds later when the next chunk arrives. In one of three turn-taking runs, only a single 60-millisecond frame was released at that moment, starving the board for 0.13 seconds. Replaying the first segment by sending audio as soon as it arrives eliminates that stutter and emits the first frame 0.19 seconds earlier.

How to Fit It into 8GB of VRAM

TTS runs on a patched C++ engine based on qwen3-tts.cpp. Two patches matter most: deduplicating Q/K/V weights saves 488 MiB of VRAM, and keeping the text embedding table on the CPU saves 594 MiB. With both models loaded simultaneously, the GPU shows 7,448 MiB used out of 8,188 MiB; about 6.8GB of that is the two models, with the rest being the desktop environment.

Q/K/V weight deduplication is a practical technique in Transformer inference optimization. In multi-head attention, the query (Query), key (Key), and value (Value) projection matrices sometimes share some parameters or have structural redundancy. By identifying and merging these duplicate or shareable weight tensors, VRAM usage can be reduced with almost no loss of accuracy. On larger models (such as the Qwen3-TTS 1.7B in this article), such optimizations often free up hundreds of MiB of precious VRAM, enabling multi-model coexistence.

Placing the text embedding table on the CPU is a common strategy in hybrid inference. Transformer models usually contain a huge word embedding matrix (Embedding Table) whose parameter count may rival the model body, but at inference time each token performs only a single table lookup—a classic "memory-access-intensive but computationally sparse" operation. Keeping this portion of the weights in CPU memory, with the GPU only reading the corresponding embedding vectors when needed, can significantly reduce peak VRAM usage. The cost is a small amount of PCIe transfer latency, but in scenarios where text generation is far faster than speech playback, this overhead is usually negligible. It is precisely through these two optimizations that this article gets Gemma 4 E4B and Qwen3-TTS 1.7B to reside simultaneously in 8GB of VRAM.

Limitations and Unfinished Business

The entire system is fully self-hosted. Testing was done on only a single machine (RTX 4060 Laptop, Windows 11), and it was never run on Linux or macOS. Tool calling is not yet supported on the LLM side. No dedicated board is required—it can also run as an HTTP service for a local LLM+TTS.

The code, raw data measurements, PCM traces, and troubleshooting notes have been open-sourced under the Apache-2.0 license. The developer poses a question to the community: if you run an LLM and TTS on the same consumer GPU, does turn-taking still beat parallelism?

One-line Summary

On an 8GB-VRAM GPU, having the LLM and TTS take turns instead of running in parallel cuts first-audio latency from 3.9s to 2.3s and nearly eliminates playback stuttering.

Paragraph Summary

While building a local offline voice assistant on an RTX 4060 Laptop (8GB VRAM), a developer found that running the LLM and TTS in parallel caused severe lag in first-sentence TTS synthesis, draining the playback buffer and causing stuttering. PCM tracing showed that stutters were concentrated in the first 13 seconds of LLM generation, as TTS's small frame-by-frame decoding kernel calls were delayed queuing in the GPU compute queue. The developer switched to turn-taking scheduling: the LLM pauses after writing one sentence, TTS synthesizes at full speed, and then the LLM continues. Results showed first-audio latency dropping from 3.9s to 2.3s and stutters from an average of 2.84s to 0.09s, while weight deduplication and placing the embedding table on the CPU let both models coexist stably within 8GB of VRAM. The solution has been open-sourced under the Apache-2.0 license.