llamadart Local Voice in Practice: Giving Llama Village a Voice

In Llama Village, a virtual world driven by local AI, conversations between characters have always stayed at the text level. In the third article of his series, developer gde tries to give this village a voice using llamadart—letting the character Dash fly over to ask Mo, "Have you seen Pip's red scarf?", and having Mo reply with speech. This experiment demonstrates a concrete path for running the full pipeline of "voice input → dialogue generation → voice output" entirely with local models on macOS.

Two Implementation Routes for the Voice Pipeline

The article presents two technical routes from the player's voice to a character's spoken reply. Both end with TTS; the difference is how audio is handled in between.

The first is via text: Qwen3-ASR first transcribes the recording into text, the dialogue model generates a reply based on the text, and then Qwen3-TTS synthesizes the speech. The advantage of this route is that the transcription is preserved, so it can be investigated and corrected when a proper noun like "Pip" is misheard.

The second is direct audio: Qwen2.5-Omni takes the recording directly and writes a reply, with no separate transcription step. However, the GGUF-converted version of Omni can only accept audio, not generate it, so Mo's reply still has to go through TTS.

Both routes use the same 3B dialogue model for comparative experiments. In the first route the model receives text; in the second the model additionally loads an audio projector. The author emphasizes that this is a multimodal route that was actually tested, rather than a claim that all multimodal models require a separate TTS.

To control size, the article also tests a smaller model pair: Qwen3.5 0.8B for dialogue, Qwen3-TTS 0.6B Base for speech, and Qwen3-ASR 0.6B still handling recognition. The small-model pair is the default configuration, while staged mode keeps the original 3B/1.7B combination for comparison.

What are Qwen3-ASR / Qwen3-TTS / Qwen2.5-Omni: This family of models comes from Alibaba's Tongyi Qianwen team, and each handles a different part of the voice pipeline. ASR (Automatic Speech Recognition) transcribes audio into text; TTS (Text-to-Speech) synthesizes text into audio, and Qwen3-TTS belongs to a class of models that support zero-shot voice cloning, allowing a speaker to be specified from a short reference audio clip; Omni is a natively multimodal model that can accept text, image, and audio inputs within the same model, and the GGUF-converted version of Qwen2.5-Omni currently retains only the audio input capability, while the output side still needs TTS.

The relationship between GGUF and llamadart: GGUF is the model file format used by the llama.cpp ecosystem. It packages weights, tokenizer, and metadata into a single file, with quantization optimizations for mixed CPU/GPU inference. llamadart is a Dart binding for llama.cpp, allowing Dart programs to directly load GGUF models and manage model downloading and caching through a unified load() / download, which is why the article doesn't need extra curl scripts to prepare model files.

Running the Complete Example on a Mac

The entire example requires macOS and Dart 3.10.7 or newer, with five small model files taking up about 2.31 GB. The author tested it on an M4 Max (64 GB memory) with Dart 3.13.5 and llamadart 0.11.0. The commands rely on macOS's built-in say, afconvert, and afplay; this is a command-line program, not a phone or browser demo.

Project Setup and Program Structure

Creating the project takes only three steps: create a new directory and configure pubspec.yaml to depend on llamadart 0.11.0, run dart pub get, and then write the main program. The program uses a fixed ModelSource.huggingFace reference, and each engine's load() method pulls missing files on demand through llamadart's download manager and reuses the cache, with no need for extra curl scripts.

The program loads only one engine from start to finish, releases it when done, and then loads the next. The core calls comprise three steps: asr.transcribeOnce() turns the recording into text, llm.create() generates a reply, and tts.synthesizeOnce() synthesizes the text into a WAV file.

Testing the Full Pipeline with a Synthesized Question

No microphone package or real recording is needed; generating a synthesized question with macOS say is enough to test the full path:

say -o question.aiff "Mo, have you seen Pip's red scarf?"
afconvert -f WAVE -d LEI16@16000 -c 1 question.aiff question.wav
dart run bin/voice.dart question.wav small
afplay reply.wav

The program prints the transcribed text and the model's reply, and saves the reply as reply.wav. When switching to audio mode, Omni's audio model and projector are downloaded and loaded, along with the original 1.7B TTS combination. This route skips ASR and sends LlamaAudioContent directly to the dialogue model. The audio route totals about 5.13 GB of files, and caching all three modes requires about 7.44 GB.

Giving Mo a Distinctive Voice

The TTS model supports specifying a speaker's timbre through a short reference audio clip. The author's vision is to make Mo sound warm and unhurried, Dash brighter and livelier, with multiple characters sharing the same TTS model but each having a different voice. When the player plays Dash, the player's own voice naturally serves as Dash's timbre.

The program's third optional argument is the reference audio: small means the 0.8B/0.6B combination, and staged means the original 3B/1.7B combination. The reference audio is passed as speakerReference to TextToSpeechRequest. In testing, the author synthesized a reference clip using macOS's Daniel voice.

Qwen's voice-design workflow also allows creating a voice during development and then using the generated clip with the Base model. Players only need to download the synthesis model and the reference audio, not the voice-design model itself, though the author did not test this path.

There is a capability boundary worth noting here: the currently tested API cannot achieve instruction control like "say this sentence in a nervous tone." This is not a matter of missing Dart parameters—Qwen's 1.7B CustomVoice and VoiceDesign models support instruction control, but the Base model used in this article does not offer that mode. To expose this capability, a compatible model, native runtime support, and the Dart API are all required. Modifying the dialogue prompt can only change the text content; it cannot change the tone.

Reference audio and speakerReference: TTS zero-shot voice cloning does not train a new model; instead, it encodes a few seconds to a dozen seconds of clean human speech into a speaker embedding, which is injected as a condition during synthesis. Thus the same weights can "play" any number of characters, as long as each provides its own reference audio. The quality of the reference audio is critical: single speaker, no background music, no reverb, otherwise the timbre will pick up noise. In the article, the macOS say-synthesized Daniel voice is used as reference, which amounts to guiding the model with synthetic speech and introduces a slight "second-hand" timbre loss; real recordings usually yield more natural results.

Instruction control: Qwen's CustomVoice and VoiceDesign models support attaching instructions about emotion, tone, speaking rate, and so on beyond the text, for example "say it in a nervous tone." This capability depends on the model having seen instruction-speech paired data during training. The Base model lacks this capability, so even if the Dart side passes instruction parameters they have nowhere to take effect; it can only be used when the model, the native runtime, and the Dart API all support it at the same time.

Are Smaller Models Faster? Measured Data

The author compared the small-model pair and the original model pair on the same M4 Max. Both kept Qwen3-ASR 0.6B and used the same synthesized question and Daniel voice reference.

Text route	Model files	Median full turn	Peak process RSS
Omni 3B + TTS 1.7B	4.61 GB	4.27 s	3.86 GiB
Qwen3.5 0.8B + TTS 0.6B	2.31 GB	5.83 s	2.51 GiB

The result is somewhat counterintuitive: download size halved, peak memory down about 35%, but in this test the full turn was actually slower.

One reason is that the smaller model's replies are longer. The 3B model says "Yes, Pip's red scarf is in the reeds by the pond," while the 0.8B model says "Yes, I saw it! It's stuffed in the reeds by the pond," ignoring the instruction to answer in one sentence. More words mean more speech to synthesize.

The test data for separated components is clearer:

Loaded component	Median generation time	Peak process RSS
Omni 3B dialogue	171 ms	2.57 GiB
Qwen3.5 0.8B dialogue	96 ms	1.08 GiB
Qwen3-TTS 1.7B	1.78 s	3.85 GiB
Qwen3-TTS 0.6B	1.78 s	2.48 GiB

The dialogue stage is indeed faster with 0.8B (96 ms vs 171 ms), and memory is much lower. But the TTS stage has the same median generation time for both, and the 0.6B synthesized audio is longer (4.80–6.48 s, versus 4.16–4.56 s for 1.7B).

Did the Models Get the Facts Right?

Both dialogue models were correct about the scarf's location and color. But when asked "Who put the scarf in the pond?", both models blamed Pip—a fact that was never provided, even though the prompt said "don't make up facts you don't know." The smaller model's "I saw it" also added an unsupported claim about how Mo came to know.

On the TTS side, the speech in all six measured clips could be correctly transcribed back to the original sentences by Qwen3-ASR, showing that intelligibility is fine, but this does not equal a score for naturalness, emotion, or similarity to Daniel's timbre. The author suggests listening to more sentences before settling on a character voice.

The author's conclusion is: in this prototype he would choose the small-model pair to control the storage budget; he would put the game's knowledge rules outside the prompt; and he would actually test latency in real dialogue scenarios before claiming an advantage.

Why dialogue is faster but the full turn is slower: The full-turn time is roughly ASR time + dialogue generation time + TTS synthesis time. 0.8B is indeed faster in the dialogue stage (96 ms vs 171 ms), but the TTS stage has a similar median generation time for both, and TTS duration is approximately linear with the length of the output text. The 0.8B ignored the "answer in one sentence" instruction and generated longer sentences, so all the extra words turned into extra synthesis time, offsetting and then exceeding the time saved in the dialogue stage.

What peak RSS means: RSS (Resident Set Size) refers to the number of pages currently actually resident in physical memory for a process, including model weights, KV cache, and runtime buffers. Because the program loads ASR, the dialogue model, and TTS in sequence and releases them after use, peak RSS mainly reflects the largest single engine at that time, not the sum of all three; this is also why the total model file size (2.31 GB / 4.61 GB) and peak memory (2.51 GiB / 3.86 GiB) are not exactly consistent.

Engineering Trade-offs for Game Integration

The TTS API currently returns the complete audio before playback begins; progress events are not playable audio chunks. The author's suggestion is: first display Mo's reply immediately as text, and play the voice after synthesis is complete.

For the game's first version, the author leans toward strategies such as push-to-talk, stopping recording while Mo speaks, and discarding old replies when the player starts a new turn. The text path needs to remain available when voice is off or the model cannot be loaded. Before integrating voice into the game, it is also necessary to measure performance when voice and rendering run at the same time.

The whole experiment reveals a pragmatic judgment: local voice is technically already working, but to make it a good experience in a game, three things need to be handled—model capability boundaries, knowledge constraints, and latency experience—and none of these three can be solved by prompts alone.

One-line Summary

llamadart runs the full local voice pipeline on macOS and reveals the trade-off that the small-model pair is faster but the full turn is slower.

Paragraph Summary

The article documents a complete experiment using llamadart on macOS to add local voice to Llama Village characters, running the pipeline of "voice input → dialogue generation → voice output" and comparing two routes: via text transcription and direct audio processing. The author measured the small-model pair of Qwen3-ASR, Qwen3.5 0.8B, and Qwen3-TTS 0.6B against the original 3B/1.7B combination: the small-model pair halves size, cuts peak memory by about 35%, and is faster in the dialogue stage, but because it ignores the "answer in one sentence" instruction and generates longer text, the full turn is actually slower. TTS supports specifying a character's timbre with reference audio, but the Base model does not support tone instruction control. In addition, both dialogue models fabricated information when facts were not provided, showing that knowledge constraints must be placed outside the prompt; the TTS API also must wait for the entire audio to be generated before playback, so game integration should display text first and play voice afterward.