Marimba: A Unified Model for Typed Speech Decisions Driven by Frozen ASR Features

Marimba unifies multiple speech decision tasks with a frozen ASR backbone plus lightweight readout layers.
Marimba proposes using frozen Qwen3-ASR 0.6B as a shared backbone, stacking lightweight readout interfaces on features from different depths to unify word presence detection, turn-end detection, language identification, speaker change detection and counting, and PII detection into one system. Lexical questions are scored zero-shot directly on transcripts; non-lexical tasks use question-conditioned readout heads; language ID reuses ASR language scores; speaker-related readouts come from layer 12, and turn-end from layer 18. Public and private test sets show usable performance on most tasks, with PII errors almost entirely from ASR transcription. On turn-end detection, dedicated Smart Turn v3.2 clearly leads on its own test set, with no significant difference only on the ICSI meeting subset. Its value lies in covering multiple tasks with one structure and reducing deployment cost, not in pursuing SOTA on every point.
Speech interaction systems often need to answer several seemingly different questions within the same audio window at once: whether a certain word appeared, whether the sentence has finished, which language is being spoken, and whether the speaker has changed. The traditional approach is to train a separate model for each task, resulting in a long engineering pipeline and high maintenance costs. Marimba proposes a different idea — using a single frozen speech recognition backbone paired with lightweight readout interfaces to unify multiple "typed speech decisions" into one system.

Core Architecture: Frozen Backbone + Task-Specific Readouts
Marimba's foundation is a frozen Qwen3-ASR 0.6B feature extractor. "Frozen" means the backbone parameters are not updated at all during training, and all task adaptation happens in readout layers on top of the backbone. The direct benefits of this design are: multiple tasks share the same acoustic and linguistic representation, there is no need to repeatedly train large models for each task, and inference requires only a single backbone forward pass.
The system's readout logic follows three paths by task type:
- Lexical questions (such as whether a word appears in the audio) are scored zero-shot directly on the ASR transcript text, requiring no additional classifier training;
- Non-lexical questions (such as whether a turn has ended) use a question-conditioned readout head, with turn-end detection additionally reading from layer 18 features;
- Language identification reuses the ASR model's built-in language scores, adding no extra modules.
In addition, speaker-related readouts are extracted from layer 12 features, used to detect speaker changes and count speakers. This "one layer, one readout" allocation suggests that different depths of the backbone carry different levels of information, and the authors clearly arrived at these choices through layer-by-layer probing.
Frozen Backbone is a common strategy in transfer learning: keeping all parameters of a pretrained model (here Qwen3-ASR 0.6B) unchanged and training only small readout layers on top of its output features. This has three advantages: first, it avoids catastrophic forgetting caused by fine-tuning large models on small task datasets; second, multiple tasks share the same forward computation, so inference cost does not grow linearly with the number of tasks; third, readout layers have few parameters, train quickly, and are easy to iterate on independently. The cost is that the backbone's representational capacity becomes the performance ceiling — if the information required for a task is not sufficiently captured by the pretraining objective, the readout layer cannot compensate. Marimba's choice of freezing rather than fine-tuning is essentially an engineering trade-off between multi-task unification and single-task optimality.
AUROC (Area Under the Receiver Operating Characteristic Curve) is the most commonly used threshold-independent metric in binary classification and detection tasks, where 0.5 is equivalent to random guessing and 1.0 is perfect discrimination. It measures the probability that the model scores a positive sample higher than a negative sample, so it is unaffected by the choice of decision threshold and is suitable for comparing the intrinsic discriminative ability of different models. However, a high AUROC does not guarantee good calibration — a model may rank correctly while its scores deviate from true probabilities, which is exactly why the paper also reports ECE (Expected Calibration Error). ECE measures the average deviation between predicted confidence and actual accuracy; a lower value means the model "when it says it is 80% confident, it is indeed correct about 80% of the time." In product scenarios that require setting thresholds, calibration quality often has a more direct impact on user experience than AUROC.
Performance: Layered Validation on Public and Private Sets
Marimba provides concrete numbers across multiple tasks. The decoder achieves AUROC of 0.80–0.92 on a private conversational test set and 0.92–0.97 on public read speech, indicating that conversational scenarios are significantly harder than read scenarios. Speech Commands four-class accuracy is 0.93, and ECE for binary word-presence detection is at most 0.08, indicating reasonable calibration.
Personal information detection follows a different path: greedy transcripts are fed into a private curated PII pipeline, achieving entity-level F1 of 0.904 on synthesized Brazilian Portuguese speech and 1.000 on reference transcripts. This comparison reveals a key fact — errors in the end-to-end pipeline mainly come from the ASR transcription itself, and the downstream PII module is nearly perfect when given clean text.
Speaker change detection with layer-12 readouts achieves AUROC of 0.966–0.980 on AMI and VoxConverse, higher than the ECAPA baseline; speaker counting (0/1/2/3+) accuracy also exceeds pyannote 3.1.
Entity-level F1 is the standard evaluation method for information extraction tasks: a prediction counts as correct only when the predicted entity boundaries and types exactly match the annotation, and precision and recall are then computed and harmonically averaged. It is stricter than word-level or character-level metrics because boundary misalignment is judged as an error. The comparison in the paper — "0.904 on synthesized Brazilian Portuguese speech, 1.000 on reference transcripts" — is essentially replacing the input text source for the same PII module: the former input comes from greedy ASR transcription, and the latter from human reference transcripts. The gap between them is the loss propagated from ASR transcription errors to downstream tasks, which also explains why the authors assert that the bottleneck of the end-to-end pipeline lies in transcription rather than PII recognition itself.
Turn-End Detection: Still a Clear Gap
The most noteworthy result is turn-end detection. Marimba's readout achieves AUROC of 0.930, but on Smart Turn's own test set, the dedicated model Smart Turn v3.2 achieves 0.982 — a considerable gap. However, on a subset of ICSI meetings annotated by next speaker, the two achieve 0.716 and 0.712 respectively, with paired confidence intervals including zero, meaning the difference is not significant.
Reading these two results together points to a reasonable explanation: Smart Turn v3.2's advantage on its own test set may partly come from domain matching, while in the ICSI scenario, which is closer to real multi-party conversation, the gap between the general readout and the dedicated model is erased. The authors honestly report that the confidence interval includes zero, which is an honest stance, but it also shows that the general approach has not yet fully surpassed dedicated models on the turn-end task.
Turn-End Detection is a semantic upgrade of Voice Activity Detection: VAD only determines "whether someone is speaking," while turn-end detection must determine "whether the current speaker has finished speaking and it is the other party's turn." This is crucial for voice assistants — interrupting too early truncates the user's speech, while responding too late feels sluggish. The difficulty lies in the interplay of prosody, syntactic completeness, and pragmatic habits, and the distribution of end-of-turn cues varies greatly across corpora (read speech, telephone, multi-party meetings). This also explains why the dedicated model Smart Turn v3.2 leads by a large margin on its own test set but is nearly on par with the general readout on the ICSI multi-party meeting subset: the gains from domain matching shrink significantly when crossing domains, and the paired confidence interval including zero is precisely the statistical manifestation of this shrinkage.
The Value and Boundaries of the Unified Paradigm
Marimba's significance is not that it is the strongest at every task, but that it covers multiple types of decisions — word presence, turn-end, language, speaker change and counting, and PII detection — with a single frozen backbone plus lightweight readouts. For product teams that need to handle multiple speech judgments simultaneously, this means fewer model assets and a simpler deployment pipeline.
The boundaries are equally clear: lexical tasks depend on ASR transcription quality, and PII detection errors almost entirely come from the transcription stage; turn-end detection still lags on dedicated test sets; and although speaker counting outperforms pyannote 3.1, the specific error distribution is not elaborated in the original text. This system is more of an engineering solution serving the goal of "good enough and unified" rather than pursuing SOTA on every single point.
For teams building speech interaction pipelines, Marimba offers a worthwhile compromise: if you are already using Qwen3-ASR, then stacking readout layers on its frozen features may be a realistic path to obtaining multi-task speech decision capabilities at low cost.
Related articles

Abnormal Error Rate Spike in Claude Mythos 5.2 and Fable 5.2
Claude Mythos 5.2 and Claude Fable 5.2 show an abnormal spike in request error rates, marked as investigating on the official status page. This article covers the incident overview, practical impact on developers, and recommended responses.

ReAct Runtime Context Trimming: Why Claude Code Acts at 70%
A deep dive into ReAct runtime context trimming: why Claude Code compresses at ~70% usage, why tool calls dominate, why messages must be dropped in pairs, and why codebases don't need RAG.

AI Agent vs LLM: How Cursor and Claude Code Actually Work
An in-depth analysis of the essential difference between AI Agents and LLMs: LLMs generate code suggestions, while Agents modify files and run tests via tool calls. Using a Spring Boot bug fix as an example, it breaks down the workflows, permission control, and verification mechanisms of Cursor and Claude Code.