LLM Tool Misselection in Practice: Using the pre_gateway_dispatch Hook to Fix Voice Memo Skill Routing

When building voice-driven automation systems, a deceptively simple problem often drives developers mad: the model receives the right instruction but calls the wrong tool. This debugging record from real engineering practice shows how to fix a voice-memo skill being misrouted to a built-in memory tool in Hermes Agent — the core technique is bypassing prompt engineering and intercepting messages directly at the gateway layer.

The Problem: Transcription Works, Skill Never Fires

The entire system runs on an NVIDIA DGX Spark with Hermes Agent (v0.20.6) and the Qwen 3.6-35B model. The expected flow: a user sends a voice memo via Telegram, a local Whisper service transcribes it, and the Agent formats the text into a structured document saved to /sandbox/notes/memos/.

The problem emerged at skill activation. In the second round of testing, despite all voice transcriptions succeeding, the voice-memo skill activation rate was 0/3. The transcribed text correctly began with the Korean word 메모 (meaning "memo"), but the Qwen model consistently chose the default memory tool to store the information instead of invoking the dedicated voice-memo skill. As a result, all notes were written to the generic MEMORY.md file and never processed into structured Markdown documents in the target directory.

Two Failed Attempts

The author first tried the most intuitive approach — modifying the skill activation condition to trigger when transcribed text begins with 메모. It failed: the model still prioritized the memory tool, ignoring the activation condition entirely.

The second attempt targeted transcription quality, adding technical vocabulary hints to the Whisper STT engine to reduce transcription errors. This partially worked, reducing errors from 6 to 3 in specific test cases. But it only improved input quality without addressing the core issue of tool selection.

The shared lesson from both attempts: when the root cause lies in the model's semantic understanding, tuning inputs or tweaking conditions treats the symptom, not the disease.

Root Cause: Semantic Confusion at the Model Level

The real problem was semantic confusion at the model level. When the Agent received a message beginning with 메모, the Qwen model interpreted the intent as "remember this," and invoked the built-in memory tool.

The key issue: Hermes v0.20.6's gateway passes the transcribed text to the model as a plain quoted string, with no voice markers or source metadata attached. This means the model has no way to distinguish between two scenarios: casually mentioning the word "memo" in conversation versus issuing a formal command to trigger the voice-memo skill. Without contextual signals, the model can only guess based on literal semantics — and it guesses wrong.

This is a classic problem in tool calling scenarios: when functionally similar tools coexist, the model's selection depends heavily on the clarity of input signals, not the developer's good intentions.

Background: Tool Calling / Function Calling Tool Calling (also called Function Calling) is the core mechanism of modern LLM Agent architectures: when generating a response, the model can output a structured "call intent," which the framework executes against the actual function and returns results back to the model. The side effect is that tool selection depends heavily on the model's semantic interpretation of the input rather than hardcoded routing rules. When semantically similar tools exist in the same system — such as "record a memo" and "invoke the voice-memo skill" being nearly identical in literal meaning — the model will "guess" the most appropriate tool based on the context probability distribution during inference. If the input signal isn't sufficiently distinctive, the model tends to select the general-purpose tool it has seen more often in pretraining data (like memory) rather than a specialized skill designed for a specific scenario. This selection bias is fundamentally a probability problem and cannot be fully eliminated simply by adjusting activation conditions or optimizing upstream inputs.

The Fix: The pre_gateway_dispatch Hook

The author's solution was to write a custom Hermes plugin using the pre_gateway_dispatch hook. This hook intercepts messages before they reach the model:

• For audio messages, it performs a lightweight pre-transcription (~0.7 seconds) to check whether the first word is 메모;
• For text messages, it checks whether the text begins with 메모:.

Once a match is found, the plugin rewrites the message, injecting a hardcoded instruction that explicitly tells the model: this is the author's memo — do not use the memory tool — instead call the skill named voice-memo and follow it exactly. The core logic:

SPOKEN = re.compile(r"^\W*메모(?=[\s,.:!?~…]|$)")
TYPED = re.compile(r"^\s*메모\s*[:：]")
INSTRUCTION = ("[voice-memo] This message is the author's memo (it starts with the word 메모). "
               "Do not use the memory tool and do not continue earlier topics. "
               "Call skill_view with name \"voice-memo\" and follow it exactly...")

def on_dispatch(event=None, **_):
    try:
        kind = getattr(getattr(event, "message_type", None), "value", "")
        text = (getattr(event, "text", "") or "").strip()
        if kind == "text" and TYPED.match(text):
            return {"action": "rewrite", "text": f"{INSTRUCTION}\n\n{text}"}
        media = list(getattr(event, "media_urls", None) or [])
        if kind in ("voice", "audio") and media and SPOKEN.match(_transcribe(media[0]).strip()):
            return {"action": "rewrite", "text": f"{text}\n\n{INSTRUCTION}" if text else INSTRUCTION}
    except Exception as e:
        logger.warning("voice_memo: 판단 실패 %s", e)
    return None

Hook registration is equally straightforward:

def register(ctx):
    ctx.register_hook("pre_gateway_dispatch", on_dispatch)

Note the exception handling design — even if the hook's judgment fails, the message passes through unchanged, ensuring zero interference with normal conversation flow. This "fail-safe degradation" pattern is a fundamental discipline for production-grade plugins.

Background: The Hook Mechanism Hooks are a classic pattern in middleware and plugin architectures: the framework reserves "slots" at key points in the processing pipeline, allowing external code to inject custom behavior without modifying core logic. pre_gateway_dispatch — the "pre-gateway-dispatch" hook — fires after source validation is complete but before the message enters LLM inference, placing it precisely in the window where message content can be safely rewritten. Compared to adding soft instructions in the System Prompt, hardcoded injection at the gateway layer has a decisive advantage: it is synchronous and deterministic, independent of the model's willingness to comply with prompt instructions. This embodies the principle of Defensive Engineering — imposing constraints on the physical data path rather than relying on the model's "understanding intent." Similar interception mechanisms are widely used in content filtering, permission checks, and request rewriting, and are a common architectural pattern in production-grade Agent systems.

Result: 3/3 Success, Zero memory Tool Calls

In the third round of testing, the fix worked completely: document save success rate reached 3/3, the memory tool was called 0 times, and all notes correctly landed in the target directory as structured documents.

The improvement from 0/3 to 3/3 validated a key insight: when prompt engineering repeatedly fails, injecting hard constraints at the data flow layer is often more reliable than continuing to coax the model.

Lessons Learned and Remaining Issues

The author distilled two reusable takeaways:

• When an LLM consistently misroutes a command to the wrong tool, don't rely solely on prompts — use a pre-dispatch hook to inject explicit constraints;
• For voice-driven automation, performing a lightweight pre-transcription at the gateway layer provides the critical context needed to guide model behavior.

Some issues remain unresolved. One is interrupt handling: if a new voice message arrives while the Agent is busy (busy_input_mode: interrupt), the current session gets interrupted. The other is transcription hallucination — Whisper occasionally appends phantom content like "Thank you" at the end of silent segments.

These remaining issues remind us that every link in a voice automation pipeline (STT, gateway, model, concurrent scheduling) can be a source of error, and end-to-end stability requires coordinated governance at multiple levels.

Note: The original post was written by the author's local Gemma model based on commit logs and experiment notes, then cross-verified by a local Qwen3.6 model, with no human editing. This meta-detail itself reflects an emerging practice: using local LLMs to assist with engineering documentation.