Marimba: How a 0.6B Model Uses a Single Encoder for Speech Question Answering

Marimba uses a frozen encoder plus lightweight readout layers for low-cost real-time speech QA.
Charcot-AI's Marimba is a lightweight perception model for voice interaction, combining a frozen Qwen3-ASR-0.6B encoder with multiple lightweight readout layers to answer typed questions like turn-end, language, and keyword detection in a single encoding. Positioned as a System One model, it streams the last 8 seconds every 250 ms, runs locally on CPU/GPU, and is Apache-2.0 open source. On 1000 paired public questions, it beats several large audio LLMs on term and turn-end detection with AUROC 0.935 (in-distribution), but underperforms on Speech Commands, where Smart Turn v3.2 remains better. Marimba integrates with Pipecat, LiveKit, and Vapi as a turn detector.
In voice interaction, we often need to answer some simple but critical questions: Is the other person still speaking? Has this sentence finished? Which language is being spoken? Was a certain keyword mentioned? Traditional solutions often require stacking multiple specialized models or calling large audio LLMs, both of which come with high latency and cost. Marimba, launched by Charcot-AI, offers a new approach: using a frozen Qwen3-ASR-0.6B encoder paired with lightweight readout layers to answer all questions at once.
Design Philosophy: System One-style Speech Understanding
Marimba is positioned as "Jev for audio"—just as Jev handles certain vision tasks in a minimalist way, Marimba pursues fast, direct, and low-cost speech perception. It is described as a System One model: it does not perform complex reasoning chains but instead gives typed answers—yes/no, single-choice or multiple-choice, calibrated scores—for the most recent 8-second audio window.
This design targets real-time interaction scenarios. The model processes audio in a streaming fashion every 250 milliseconds, can run locally on CPU or GPU, and is open-sourced under the Apache-2.0 license. Installation is via pip install "marimba-audio[torch]", and usage is very straightforward:
from marimba import Marimba, Question
model = Marimba.from_pretrained(backend="torch", device="cpu", dtype="float32")
questions = [
Question.score("Has the speaker finished?"),
Question.choice("What language?", ["en", "pt", "es"]),
]
for d in model("clip.wav", questions):
print(d.task, d.value, d.probability)
The core idea is encode once, read many times. After the frozen encoder extracts audio features, different "small readout layers" are responsible for answering different questions. This means the marginal cost of adding new questions is very low—there is no need to retrain the entire model.
System One / System Two is a term borrowed from psychologist Daniel Kahneman's dual-system cognitive theory and applied to AI. System One refers to fast, intuitive, automatic judgments that consume almost no explicit reasoning; System Two refers to slow, deliberate thinking processes that require multi-step reasoning. In model design, System One-style models usually refer to small models that give answers in a single forward pass or in very few steps, while System Two-style systems rely on chain-of-thought, tool calls, or multi-round planning. Marimba positions itself as the former, emphasizing its low latency and low compute overhead—it does not aim to explain the reasoning process, only to give typed judgment results at millisecond speed.
Frozen encoder + lightweight readout layers is a common parameter-efficient transfer paradigm. The encoder (here Qwen3-ASR-0.6B) keeps its weights unchanged during training and serves only as a fixed feature extractor; the readout head is a newly trained small classification or regression network attached to the encoder output. This has three advantages: first, training cost is extremely low because only a small number of parameters need optimization; second, multiple readout layers can share the same audio features, achieving "encode once, read many times"; third, when adding new question types, only new readout layers need training, without touching the base model. The trade-off is that the encoder's representational capacity determines the performance ceiling—if the information required for a certain task is not captured by the encoder, even a strong readout layer cannot compensate.
Performance: Small Models Can Punch Above Their Weight in Some Areas
On 1000 paired public questions, Marimba was compared with larger audio LLMs, including Qwen3-Omni-30B-A3B, Qwen2-Audio-7B, and FireRedAudio 10.6B. The results show that Marimba wins in binary term detection on Common Voice and FLEURS, and leads by a wide margin in turn-end detection—achieving an AUROC of 0.935, while the comparison models range from 0.54 to 0.79.
It is important to view this objectively: turn-end detection is an in-distribution task for Marimba because the training data includes Smart Turn data. Even so, it performs poorly on Speech Commands, and Smart Turn v3.2 still outperforms Marimba on its own test set (0.982 vs. 0.930). These results indicate that Marimba is not a jack-of-all-trades; its strengths are concentrated in specific types of speech judgment tasks.
AUROC (Area Under the Receiver Operating Characteristic curve) is a common metric for evaluating the ranking ability of binary classification models, ranging from 0 to 1. It measures the probability that, given a randomly chosen positive sample and a randomly chosen negative sample, the model assigns a higher score to the positive sample. 0.5 corresponds to random guessing, and 1.0 indicates perfect separation. Thus, 0.935 means the model's judgments are in the correct direction for the vast majority of sample pairs, while 0.54 to 0.79 indicates that the comparison models are near or slightly better than random at distinguishing "finished speaking" from "still speaking." Note that AUROC only reflects ranking quality; it does not directly give accuracy at a specific threshold. In actual deployment, the decision threshold still needs to be chosen based on business requirements.
In-distribution tasks refer to test data that comes from the same or a highly similar data distribution as the training data. When a model has seen similar data during training for a task, its performance on that task is usually significantly better than on unseen tasks, so such results cannot be directly extrapolated to open scenarios. To evaluate whether a model is general-purpose, more attention should be paid to its performance on out-of-distribution tasks—for example, Marimba's failure in keyword detection on Speech Commands mentioned in the article.
Application Scenarios: A Plug-and-Play Turn Detector
Marimba can already be integrated into Pipecat, LiveKit, and Vapi as a turn detector. For developers building voice assistants or real-time dialogue systems, this means a single lightweight model can replace combinations of multiple models or cloud services that might have been needed before.
Local execution, streaming processing, and low latency make Marimba suitable for scenarios with privacy and response-speed requirements. At the same time, the Apache-2.0 license provides commercially friendly terms.
Turn-end detection (also called end-of-turn detection) is a key component in voice dialogue systems, used to determine whether the user has finished speaking and whether the system can start responding. Judging too early interrupts the user; judging too late makes the conversation feel sluggish. Traditional approaches rely on voice activity detection (VAD) plus a fixed silence duration threshold, but people produce pauses of varying lengths when thinking, breathing, or changing speaking rate, making a fixed threshold difficult to balance. Model-based turn detection combines prosodic, semantic, and contextual cues to output a probability score for "finished," making the dialogue system's responses more natural.
Limitations and Future Directions
Marimba's current capabilities are limited to the types of question-answering it was trained on. In the release, the authors explicitly welcome community feedback on what questions they would like the model to answer—implying that readout layers can continue to be extended, but the capability boundary of the base encoder still exists. For tasks requiring complex audio reasoning, large models remain the more appropriate choice.
The project provides an online demo, model card, code repository, and paper for researchers and developers to evaluate in depth. If you are building voice interaction applications, Marimba is worth considering as a lightweight component.
One-line Summary
Marimba uses a frozen encoder plus lightweight readout layers to enable real-time speech question answering at low cost.
Paragraph Summary
Marimba, launched by Charcot-AI, is a lightweight perception model for voice interaction. It uses a frozen Qwen3-ASR-0.6B encoder with multiple lightweight readout layers, so a single encoding can answer typed questions such as whether the speaker has finished, the language type, and keyword detection. Positioned as a System One-style model, it streams the most recent 8 seconds of audio every 250 milliseconds, can run locally on CPU/GPU, and is open-sourced under Apache-2.0. On 1000 paired public questions, Marimba outperforms several large audio LLMs in term detection and turn-end detection, achieving an AUROC of 0.935—but this task is an in-distribution result; it performs poorly on Speech Commands, and Smart Turn v3.2 remains better on its own test set. Marimba can already be integrated into Pipecat, LiveKit, and Vapi as a turn detector.
Related articles

RePlan-VLA: Progress-Aware Replanning Breakthrough for Long-Horizon Robotic Manipulation
The RePlan-VLA framework converts VLA models' short-horizon capabilities into reliable long-horizon execution through progress-aware adaptive replanning, improving success rates by 21.9% and 10.4% on RoboCasa365 and CALVIN, and validating real-world deployment on a Franka Research 3 robot.

Geometric Bridge Flow Matching: A New Breakthrough in Blind Rigid Motion Correction
Geometric bridge flow matching enables blind rigid motion correction in head CBCT, achieving 37.1 dB PSNR and 0.27 mm RPE on simulated data in ~9.1 minutes, outperforming existing methods.

Google Releases Gemini 4 Hydrogen: 2.5x Faster Without Sacrificing Intelligence
Google releases the lightweight Gemini 4 Hydrogen: 2.5x faster task completion, AA Intelligence Index of 54 beating its flagship sibling, 7.3s single-step on Agent Arena faster than Claude and GPT rivals, four thinking levels with pay-as-you-need pricing, now available across AI Studio and more.